{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s_qNSzzyaCbD"
   },
   "source": [
    "##### Copyright 2019 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "jmjh290raIky"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J0Qjg6vuaHNt"
   },
   "source": [
    "# 理解语言的 Transformer 模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AOpGoE2T-YXS"
   },
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/text/transformer\">\n",
    "    <img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\" />\n",
    "    在 tensorflow.google.cn 上查看</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/zh-cn/tutorials/text/transformer.ipynb\">\n",
    "    <img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\" />\n",
    "    在 Google Colab 运行</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/zh-cn/tutorials/text/transformer.ipynb\">\n",
    "    <img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\" />\n",
    "    在 Github 上查看源代码</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/zh-cn/tutorials/text/transformer.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\" />下载此 notebook</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7Saq5g1mnE5Y"
   },
   "source": [
    "Note: 我们的 TensorFlow 社区翻译了这些文档。因为社区翻译是尽力而为， 所以无法保证它们是最准确的，并且反映了最新的\n",
    "[官方英文文档](https://www.tensorflow.org/?hl=en)。如果您有改进此翻译的建议， 请提交 pull request 到\n",
    "[tensorflow/docs](https://github.com/tensorflow/docs) GitHub 仓库。要志愿地撰写或者审核译文，请加入\n",
    "[docs-zh-cn@tensorflow.org Google Group](https://groups.google.com/a/tensorflow.org/forum/#!forum/docs-zh-cn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M-f8TnGpE_ex"
   },
   "source": [
    "本教程训练了一个 <a href=\"https://arxiv.org/abs/1706.03762\" class=\"external\">Transformer 模型</a> 用于将葡萄牙语翻译成英语。这是一个高级示例，假定您具备[文本生成（text generation）](text_generation.ipynb)和 [注意力机制（attention）](nmt_with_attention.ipynb) 的知识。\n",
    "\n",
    "Transformer 模型的核心思想是*自注意力机制（self-attention）*——能注意输入序列的不同位置以计算该序列的表示的能力。Transformer 创建了多层自注意力层（self-attetion layers）组成的堆栈，下文的*按比缩放的点积注意力（Scaled dot product attention）*和*多头注意力（Multi-head attention）*部分对此进行了说明。\n",
    "\n",
    "一个 transformer 模型用自注意力层而非 [RNNs](text_classification_rnn.ipynb) 或 [CNNs](../images/intro_to_cnns.ipynb) 来处理变长的输入。这种通用架构有一系列的优势：\n",
    "\n",
    "* 它不对数据间的时间/空间关系做任何假设。这是处理一组对象（objects）的理想选择（例如，[星际争霸单位（StarCraft units）](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/#block-8)）。\n",
    "* 层输出可以并行计算，而非像 RNN 这样的序列计算。\n",
    "* 远距离项可以影响彼此的输出，而无需经过许多 RNN 步骤或卷积层（例如，参见[场景记忆 Transformer（Scene Memory Transformer）](https://arxiv.org/pdf/1903.03878.pdf)）\n",
    "* 它能学习长距离的依赖。在许多序列任务中，这是一项挑战。\n",
    "\n",
    "该架构的缺点是：\n",
    "\n",
    "* 对于时间序列，一个单位时间的输出是从*整个历史记录*计算的，而非仅从输入和当前的隐含状态计算得到。这*可能*效率较低。   \n",
    "* 如果输入*确实*有时间/空间的关系，像文本，则必须加入一些位置编码，否则模型将有效地看到一堆单词。\n",
    "\n",
    "在此 notebook 中训练完模型后，您将能输入葡萄牙语句子，得到其英文翻译。\n",
    "\n",
    "<img src=\"https://tensorflow.google.cn/images/tutorials/transformer/attention_map_portuguese.png\" width=\"800\" alt=\"Attention heatmap\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JjJJyJTZYebt"
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "import tensorflow_datasets as tfds\n",
    "import tensorflow as tf\n",
    "\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fd1NWMxjfsDd"
   },
   "source": [
    "## 设置输入流水线（input pipeline）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t4_Qt8W1hJE_"
   },
   "source": [
    "使用 [TFDS](https://tensorflow.google.cn/datasets) 来导入 [葡萄牙语-英语翻译数据集](https://github.com/neulab/word-embeddings-for-nmt)，该数据集来自于 [TED 演讲开放翻译项目](https://www.ted.com/participate/translate).\n",
    "\n",
    "该数据集包含来约 50000 条训练样本，1100 条验证样本，以及 2000 条测试样本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8q9t4FmN96eN"
   },
   "outputs": [],
   "source": [
    "from ipywidgets import IntProgress\n",
    "examples, metadata = tfds.load('ted_hrlr_translate/pt_to_en', with_info=True, as_supervised=True)\n",
    "train_examples, val_examples = examples['train'], examples['validation']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pt is b'e quando melhoramos a procura , tiramos a \\xc3\\xbanica vantagem da impress\\xc3\\xa3o , que \\xc3\\xa9 a serendipidade .', en is b'and when you improve searchability , you actually take away the one advantage of print , which is serendipity .' \n",
      "pt is b'mas e se estes fatores fossem ativos ?', en is b'but what if it were active ?' \n",
      "pt is b'mas eles n\\xc3\\xa3o tinham a curiosidade de me testar .', en is b\"but they did n't test for curiosity .\" \n",
      "pt is b'e esta rebeldia consciente \\xc3\\xa9 a raz\\xc3\\xa3o pela qual eu , como agn\\xc3\\xb3stica , posso ainda ter f\\xc3\\xa9 .', en is b'and this conscious defiance is why i , as an agnostic , can still have faith .' \n",
      "pt is b\"`` `` '' podem usar tudo sobre a mesa no meu corpo . ''\", en is b'you can use everything on the table on me .' \n"
     ]
    }
   ],
   "source": [
    "#查看一下部分数据\n",
    "for i,j in train_examples.take(5):\n",
    "    print(\"pt is {}, en is {} \".format(i.numpy(),j.numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RCEKotqosGfq"
   },
   "source": [
    "从训练数据集创建自定义子词分词器（subwords tokenizer）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KVBg5Q8tBk5z"
   },
   "outputs": [],
   "source": [
    "tokenizer_en = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
    "    (en.numpy() for pt, en in train_examples), target_vocab_size=2**13)\n",
    " \n",
    "tokenizer_pt = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
    "    (pt.numpy() for pt, en in train_examples), target_vocab_size=2**13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4DYWukNFkGQN"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenized string is [7915, 1248, 7946, 7194, 13, 2799, 7877]\n",
      "The original string: Transformer is awesome.\n"
     ]
    }
   ],
   "source": [
    "sample_string = 'Transformer is awesome.'\n",
    "\n",
    "tokenized_string = tokenizer_en.encode(sample_string)\n",
    "print ('Tokenized string is {}'.format(tokenized_string))\n",
    "\n",
    "original_string = tokenizer_en.decode(tokenized_string)\n",
    "print ('The original string: {}'.format(original_string))\n",
    "\n",
    "assert original_string == sample_string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o9KJWJjrsZ4Y"
   },
   "source": [
    "如果单词不在词典中，则分词器（tokenizer）通过将单词分解为子词来对字符串进行编码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bf2ntBxjkqK6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7915 ----> T\n",
      "1248 ----> ran\n",
      "7946 ----> s\n",
      "7194 ----> former \n",
      "13 ----> is \n",
      "2799 ----> awesome\n",
      "7877 ----> .\n"
     ]
    }
   ],
   "source": [
    "for ts in tokenized_string:\n",
    "    print ('{} ----> {}'.format(ts, tokenizer_en.decode([ts])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bcRp7VcQ5m6g"
   },
   "outputs": [],
   "source": [
    "BUFFER_SIZE = 20000\n",
    "BATCH_SIZE = 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kGi4PoVakxdc"
   },
   "source": [
    "将开始和结束标记（token）添加到输入和目标。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UZwnPr4R055s"
   },
   "outputs": [],
   "source": [
    "def encode(lang1, lang2):\n",
    "    lang1 = [tokenizer_pt.vocab_size] + tokenizer_pt.encode(\n",
    "        lang1.numpy()) + [tokenizer_pt.vocab_size+1]\n",
    "\n",
    "    lang2 = [tokenizer_en.vocab_size] + tokenizer_en.encode(\n",
    "        lang2.numpy()) + [tokenizer_en.vocab_size+1]\n",
    "  \n",
    "    return lang1, lang2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6JrGp5Gek6Ql"
   },
   "source": [
    "Note：为了使本示例较小且相对较快，删除长度大于40个标记的样本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2QEgbjntk6Yf"
   },
   "outputs": [],
   "source": [
    "MAX_LENGTH = 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c081xPGv1CPI"
   },
   "outputs": [],
   "source": [
    "def filter_max_length(x, y, max_length=MAX_LENGTH):\n",
    "    return tf.logical_and(tf.size(x) <= max_length,\n",
    "                        tf.size(y) <= max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Tx1sFbR-9fRs"
   },
   "source": [
    "`.map()` 内部的操作以图模式（graph mode）运行，`.map()` 接收一个不具有 numpy 属性的图张量（graph tensor）。该`分词器（tokenizer）`需要将一个字符串或 Unicode 符号，编码成整数。因此，您需要在 `tf.py_function` 内部运行编码过程，`tf.py_function` 接收一个 eager 张量，该 eager 张量有一个包含字符串值的 numpy 属性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Mah1cS-P70Iz"
   },
   "outputs": [],
   "source": [
    "def tf_encode(pt, en):\n",
    "    return tf.py_function(encode, [pt, en], [tf.int64, tf.int64])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9mk9AZdZ5bcS"
   },
   "outputs": [],
   "source": [
    "train_dataset = train_examples.map(tf_encode)\n",
    "train_dataset = train_dataset.filter(filter_max_length)\n",
    "# 将数据集缓存到内存中以加快读取速度。\n",
    "train_dataset = train_dataset.cache()\n",
    "train_dataset = train_dataset.shuffle(BUFFER_SIZE).padded_batch(\n",
    "    BATCH_SIZE, padded_shapes=([-1], [-1]))\n",
    "# 流水线技术 重叠训练的预处理和模型训练步骤。当加速器正在执行训练步骤 N 时，CPU 开始准备步骤 N + 1 的数据。这样做可以将步骤时间减少到模型训练与抽取转换数据二者所需的最大时间（而不是二者时间总和）。\n",
    "# 没有流水线技术，CPU 和 GPU/TPU 大部分时间将处于闲置状态:\n",
    "train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n",
    "\n",
    "\n",
    "val_dataset = val_examples.map(tf_encode)\n",
    "val_dataset = val_dataset.filter(filter_max_length).padded_batch(\n",
    "    BATCH_SIZE, padded_shapes=([-1], [-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_fXvfYVfQr2n"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: id=207736, shape=(64, 38), dtype=int64, numpy=\n",
       " array([[8214,  342, 3032, ...,    0,    0,    0],\n",
       "        [8214,   95,  198, ...,    0,    0,    0],\n",
       "        [8214, 4479, 7990, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [8214,  584,   12, ...,    0,    0,    0],\n",
       "        [8214,   59, 1548, ...,    0,    0,    0],\n",
       "        [8214,  118,   34, ...,    0,    0,    0]])>,\n",
       " <tf.Tensor: id=207737, shape=(64, 40), dtype=int64, numpy=\n",
       " array([[8087,   98,   25, ...,    0,    0,    0],\n",
       "        [8087,   12,   20, ...,    0,    0,    0],\n",
       "        [8087,   12, 5453, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [8087,   18, 2059, ...,    0,    0,    0],\n",
       "        [8087,   16, 1436, ...,    0,    0,    0],\n",
       "        [8087,   15,   57, ...,    0,    0,    0]])>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pt_batch, en_batch = next(iter(val_dataset))\n",
    "pt_batch, en_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nBQuibYA4n0n"
   },
   "source": [
    "## 位置编码（Positional encoding）\n",
    "\n",
    "因为该模型并不包括任何的循环（recurrence）或卷积，所以模型添加了位置编码，为模型提供一些关于单词在句子中相对位置的信息。\n",
    "\n",
    "位置编码向量被加到嵌入（embedding）向量中。嵌入表示一个 d 维空间的标记，在 d 维空间中有着相似含义的标记会离彼此更近。但是，嵌入并没有对在一句话中的词的相对位置进行编码。因此，当加上位置编码后，词将基于*它们含义的相似度以及它们在句子中的位置*，在 d 维空间中离彼此更近。\n",
    "\n",
    "参看 [位置编码](https://github.com/tensorflow/examples/blob/master/community/en/position_encoding.ipynb) 的 notebook 了解更多信息。计算位置编码的公式如下：\n",
    "\n",
    "$$\\Large{PE_{(pos, 2i)} = sin(pos / 10000^{2i / d_{model}})} $$\n",
    "$$\\Large{PE_{(pos, 2i+1)} = cos(pos / 10000^{2i / d_{model}})} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WhIOZjMNKujn"
   },
   "outputs": [],
   "source": [
    "def get_angles(pos, i, d_model):\n",
    "    angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))\n",
    "    return pos * angle_rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1Rz82wEs5biZ"
   },
   "outputs": [],
   "source": [
    "def positional_encoding(position, d_model):\n",
    "    angle_rads = get_angles(np.arange(position)[:, np.newaxis],\n",
    "                          np.arange(d_model)[np.newaxis, :],\n",
    "                          d_model)\n",
    "  \n",
    "    # 将 sin 应用于数组中的偶数索引（indices）；2i\n",
    "    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])\n",
    "  \n",
    "    # 将 cos 应用于数组中的奇数索引；2i+1\n",
    "    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])\n",
    "    \n",
    "    pos_encoding = angle_rads[np.newaxis, ...]\n",
    "    \n",
    "    return tf.cast(pos_encoding, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# np.newaxis示例\n",
    "# a=np.array([1,2,3,4,5])\n",
    "# b=a[np.newaxis,:]\n",
    "# print(a.shape,b.shape)\n",
    "# print(a)\n",
    "# print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# angle_rads = get_angles(np.arange(5)[:, np.newaxis],\n",
    "#                           np.arange(16)[np.newaxis, :],\n",
    "#                           16)\n",
    "# print(angle_rads.shape)\n",
    "# print(angle_rads)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1kLCla68EloE"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 50, 512)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEKCAYAAAD+XoUoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsnXd8HNXVhp8zs7vSqq26ZMuWe6W4YFwwzfQOoZMQSggkoXxACATyBUhIIaRASAIhkJBACj0Em8/EFAMGG2xT3I2b3GXL6tJK22b2fn/s7GolS/balmzL3Of3u96Z2ZnZu7J09+577nmPKKXQaDQazZcD40B3QKPRaDT7Dz3oazQazZcIPehrNBrNlwg96Gs0Gs2XCD3oazQazZcIPehrNBrNl4geHfRFZIOILBWRRSLyiXMsX0TeEpE1zmNeT/ZBo9FoDhQi8rSI7BCRZV08LyLyOxFZKyJLRGR80nNXO+PkGhG5urv6tD9m+tOUUmOVUhOc/buBd5RSw4B3nH2NRqM5FPkbcMYunj8TGOa0G4A/QmxyDNwPTAImAvd31wT5QMg75wPPONvPABccgD5oNBpNj6OUmgPU7eKU84FnVYyPgVwR6QOcDryllKpTStUDb7HrD4+UcXXHTXaBAt4UEQX8SSn1JFCilNoGoJTaJiLFnV0oIjcQ++QjM8N7VKvKYOyoASxauZGxI8vZ/PlyBhw2iEWbGsnM89GveRv1DSFKxx3GkjXbcHszOKzQRNkWq5pdtNbXUtS3hDLVSOX6atINoXDkQNa3CPU7ajHdHgqLcqmuqkVFo2QX5DOkwEtocwV1tQFsBbkZbrLKS2h157ChqpmS/AwK0sCq3k7LjmaarSgAGaZBZm4aaUUF2F4fWz9fjkeEzDST9Fwv7rw8ounZNIdt6lvCtAYsrFAQOxKGqM34IUVEW5oIN7cSaQkTDkcJRRW2UkQBAUwBlwgFZblYgRBW0MIO2YSjUawoiXPj+dYGkHPYSMK2ImxFCVs2YStK1FaxFo2iorbTYttjSj2Iy42YbjBMlGHGHhGiCmwFSsX6tapiGyICIgjOo2G07RsGIgYigjvNRClAKZRzj9g+qNg/xDPFlYqSlZ2OiCCAIYLzMgiCIcSec45Vbq1NvGsVu0GH38i2/SGD+iDx3zfnH3H22u/HWLl2S6q/9xw+rH+nx0V2PrZ01aaU7wtw5Mjyzu/dybHFX6R+77Fd3LczFu3BfWP3HrAH996Y+n1Htb/vopUbUYHaGqVUUco36YCR009hBVM6VwVqlwPJJz/pjHOpUgZsTtrf4hzr6vg+09OD/lSlVKUzsL8lIl+keqHzg3sS4KgjD1NLzUnMnfs4vik3MufDx/he5igee/kpCm6exTEXn8WDs3/CqzPW8P25c+l73oOUHnYU86/LxG6s5YT3C/j0pX9y+X2382D4dX789acYnuXhmpef4usL0nn1sb+RVTqQa244mz8+8i8iwRaOu+pSXv7a4ay/9ev88x9LaYxEuXBUKcf8/nssLjuJqx75gDuuGMNVg01qnvgZ8/8wh3erWwEY70tn0rnDGHLD1TQffib/mzOavmkupgz0MeKCI+l78cW0jDyJ9zc28vwnm1mypIod61bj374BK+hnwUs30Dr/Tba+v4jKhVvZuKmJDa0R6sI24ajCFPC5TQo9JlffdT41S9ZRu6qG+ooGtvrDVIds6iM2ATuK7YxxHkM48+U32dQYZENNCxtrW6isbaWlKURrY4hga5hQcwPh1kasgB8r2MLc75VjFpRi5hVDZi7RtGyi3lwiZhqtkSgtkSgBS9EUsjjp8h9huj0YLg+Gy43h8mCmeTFdnsS24fLg8rjpN6wAKxzFithYERvbimJFokStKLYdxbaiRO0otmURtcJMOXEEHpeBx2XGHk2DNJfhHGvf7r3/b6ioHfsdcj68Ytuxx6jzCPD4X3+AIWCKYIhgGrEPlY77ImAgHHXeXe3utStmvPkI0DbIx79Si3PASBqhB0y7JdU/CwDeef8PnQ7wRicHi4+7OeX7vv/hY+32O3uNOPlTb0r5vgAfzn085XNzj7kx5XPndrivb8qNRBb9NfVPjc6wgrhGnJfSqZFFfw0mSdd7Q2c/ZrWL4/tMj8o7SqlK53EH8CoxbarK+fqC87ijJ/ug0Wg0e4QIYpgptW5gC5D8tbAfULmL4/tMjw36IpIpItnxbeA0YBkwHYhHoq8GXuupPmg0Gs2eI8431t23bmA6cJWzimcy0OjI37OA00QkzwngnuYc22d6Ut4pAV51vs66gH8ppf4rIguBF0XkOmATcEkP9kGj0Wj2DGem3z23kueAE4FCEdlCbEWOG0Ap9QQwEzgLWAu0Atc6z9WJyE+Ahc6tHlBK7SognDI9NugrpSqAMZ0crwVO3pN7rdgRZsqdV/HeyElMueVRPj76eC49ophL58U+aaefm8etN67khz87mzP/OJ9gYw3P3n4c7555GpFXXmfJzF9QPuUcHjp9MO+MeI6ArTj1W1OYnz6a9994BRW1GX380bw8cxWttZUMOOZc7jplONG3/szi6aupDtmMz01n1KUTsMaezT/+u4Zx4/pw2tACogv+xYa3lrO0MUQ4qujvdTN4aB5lx4+FYZNYUR0gy2UwKNNN8RHFFE44DFV+BJuawny6uYH1W5poqqknWF+FFfQDEK5YTsPqzTSsr6dhm5/qkI3fihKOxiQ9jyFkmgb5HpOWrTW07vDTWhOgMWjht6K02LFz43q+KbFWH4hQ3xqmtiVMrT9MKGARDliEQxaRYCt2OIAdChC1wqiojZGRjZGeiXi8RF3pKE8GypVG2FKxgHBUEbajhKwoYsa/8hqIYWK4PRjOV2DD5UEME9PlQkSw49q9HUVFY4FkFVVEVexRKUU0qhLauWkIpmHEHkWc/U5aUpRURaO7/v20nXunqOe33Xf3ev6e0Flgd2/oTM+Xfbh5N3WrVyKAmN0z6CulrtjN8wroNECilHoaeLpbOpJETwdyNRqNpnchgtFNM/2DET3oazQaTQe6S945GNGDvkaj0STTjZr+wYge9DUajSYJQTBc7gPdjR6jV7hshpobeOekIG9uaWL22Savrqxm8vw5vP7Yn/npT67j7eO/ytF5Xmqu+Tnzn3+RiZdewui5j/Hqympue+wjlG1z73VHU/PQbczc2sSZ/XPo890f8/2XllCzeiHFh03l3vMOY+tn75JZ1J+zThnK5IwGlj/5Ogvrg/jcBuOnlFF04dd4e30D7y/czOUT+lPWsp6tb8xm9bJqqkIWXlMYneOh3zEDyZx0EtuNXOZurKMkzUX/gbmUThhK+hFTqPcUsGhbM59trKduWzMtOzYRbmkEwPR4CW5YR8PaSpq2NFEdsmmyYolWEAviZrkMfG4nkLu9Fn9VC611ARoj0UTA107KPDVF8BhCXTDCjqYQtf4QwUCEcCBCOGTFEqRCAaxwLIgbtSLYkTBGZg5GZjZRj5eo24typRFRxAK4UYVlQ2vEJmhFE0HbeBBXkoO4ZjyYK5guAxUlFrCNKmw76gRtVSI5SzlBXBW1UbadCNR6zFgCVjwxq2Mg1xBpF2jdVWIWdB787Io9iYnGXy+VxKy94cscZN0v7N91+vsdPdPXaDSaDvTWAT0V9KCv0Wg0yYh025LNgxE96Gs0Gk0SwqE90+8Vmn7/8j78/Jibue/3l/HwhOu4884TOPp/36LPuFO4ruo//Keinq9O/zEX/nQ2GQV9ef07k3j+pn8wPCuN9R9O58izz+PK/GpmPPoB+R6T4x+8hGfWK5a/8wGeTB+nnnE4J2bUYIcDDJ40hVuPG0jDc3/g47lb8FtRJud7GfX1aVTmH87T8zZQufILpg30EZjzKuvfXsdqfxhbwcAMD/3Hl9Jn2mSsAUfxaWUz767cwdAsN33G98E3diyRPoextj7IJxvrqdzcSPOObYT99UStMGKYeDJ91K/eTOPGJupqA4nErGTjtHhiVka+l+ZtflprA9SFbRojNkFHb09OzPIYgtc0qPOHqWsJ0xBPzArZREIWVsCPHQ4QjTh6fjw5KzMb5clEuTPAnU7UlUYwnphlK4JWlKAVpTVitzNaE8PESErKMlweDEMwTQPTNBKJWbYVRUVJaPkJbT+u6dsxXT9utGYagitJw2+n64tgOmJ3dydmSeK+qSdmdaXnd3bOvqITs7oZMTBdnpRab0TP9DUajSYZObRn+nrQ12g0miQEvU5fo9FovlQcyoN+r9D0c+q3Upru4vHh3wBgydUPsebdV5n9i7N49GuPc9Xx5TxqjWfjvBl8945LqLztayysD3LFfWeQ0284f7t+Ip/fdCeLG4Ocf9JAWs66nd88txh/1QYGTp7GD08ZyrY//pqCoeP5zrmjKN/6EYv//CErm0P097o5/Cuj8JxyFf9ZVc3yz7fRtGU13jUfsO61j1iyqZG6sE2+x2RkaSb9TxyNZ9w01jYp3ltTw9aKevoeUUzppNG4Rk9ma8jks21NLN5QR12Vn0D99sQafZc3izRfIQ1rq2ja0sT2YHyNfpvRWpYrpuf70l1klmTQUtVCc2OIxkiUYFQRsNuM2eLXeAwh3ZDEGv1QwCIUiBAJWUSCQexwbI2+7azTj2vp4s1GebwodxpRt5eQFU3o+WFb0RqxaY1ECdnRhNFa3Hgtoec7a/YN08BwGYghRK1YwRSlVKxgSgejtbjhW7zt1mjNWaNvGJLQ83e3Rj9OV3p+R1JdW7873T9+n4NVzz/QHBRd1+v0NRqN5suElnc0Go3mS4OIYLh758qcVNCDvkaj0SSjDdc0Go3my4Ue9A8w26v8XLv9E3LO+zX+BU9SeNsfmXb9dbTcchktdpTxb7zB2Rf8kiEnXsDdfSr5wd8WcdHIAoLf+BmXD6qgfM4T/Pjt9Rydl864h3/ENTNWsmHeLHzlo7j5osMpW/l/TH/qY8b++DquPLyQdbfdzocV9ZgiHDMinwFXXsqiQDbPvf851as+JWqF2THjVSrmbGJzIILHEIZneSif2o+8406kIXcIH66o5pNV1dRvraTPhIFkjp1CIH8wyzY0Mm9NDTVbm2mp3kSouT6WCOXy4MnIIaOgjIZPG6lqDlMfaQvimgJZLoMcJ5CbWZJJVkkmdWvqqQvHEriSq2tB+yCu1zSoawnR3BImFIwQDlhEQlab0ZqTmBVNCqAqT8xkTbkzsDAI21GnxYK4ITtKyLJjlbOSDNbMDkFc02VguoxYoNRlJBKxbEsljNfiiVnJRmvtArkdErPaJWg5iVnxylm7CuLGE7Og84BtnOTErL0N4na30dr+oBd0cb9g9Ib/rL2kV6ze0Wg0mv2FiCBGai3F+50hIqtEZK2I3N3J84+IyCKnrRaRhqTn7KTnpnfH++sVM32NRqPZn5hm98yHRcQEHgNOBbYAC0VkulJqRfwcpdTtSeffAoxLukVAKTW2WzrjoGf6Go1Gk4zQnTP9icBapVSFUioMPA+cv4vzrwCe64Z30SW9YtAvKfAy7lfLKTvqZE6eJZhpXt44BR57fgXfe+wKTvj1PKxgC/+550T+e/r/4DGEk176JZc9MZ+HTypm5s3PEI4qzrrzZN6WEbz16lwAxpw6hWtHZrLkwaeYU9PKT84ejf3aIyz490q2By3G56ZzxLXH0jrmHJ76eCPrP19Fa20l3rxS1s5YzOLGEAFb0TfdxbBRhfQ/ZQKMnMrn21t4c/l2qjY14K/aQNGUcUQHjWNdfYgFG+tZt6GBxqoagvVVWEE/AJ5MH968UrLzs2jY5md70Gqn0XtNI2G05stPJ6s4g8zSXBqDFo2RKC12dCejtWSztSyXUBs3WgtYhEMWkWArdjiAHQokEqKikbbEqKg7A+XJIOpOJxRPyooqwnaUkGO0Fn+Ma/iG0T45y3S5MM1YUlYsOStWQCVqO1q+k6AVT8xK1vMhppObIl0WToknVRlOgtauSNbzoevErLien8yeJg3t6g8r+V778gd4qBmtHRSJWcRdNrtt0C8DNiftb3GO7fy6IgOAQcDspMPpIvKJiHwsIhfs5Vtqh5Z3NBqNph27n0AkUSginyTtP6mUerLdzXZGdXIM4HLgZaVU8uykXClVKSKDgdkislQptS7VznWGHvQ1Go0mGUfeSZEapdSEXTy/BeiftN8PqOzi3MuBm5IPKKUqnccKEXmPmN6/T4N+r5B3NBqNZn/SjfLOQmCYiAwSEQ+xgX2nVTgiMgLIAz5KOpYnImnOdiEwFVjR8do9pVcM+oGSAax9/3WWPnwW8559hum/v54/T7qOi0YW8MaE7/D5q89xxc1XkvnYHczY0sQ3bj6GJ/1DWPTav1l76/W8vaOFrxzVB99tv+HuZz+lrmIx5RNP5dGLjqThqZ/w9vubABgXXs2nv53JwvogpekuJpwxmNyLvsm/v6jhg482Ub9hGYbLQ/7Q8SxfVcf2oIXPbXBEvpcBJ48kY8pZbIxk8s7qatatraVh82qCjdV4jjye7eQwf0sjH62poXZ7bI1+wmgtPWa0lllYSm5RBlsDFk1WdKdi6Pkek8I0F5nFmWT1zSarrIi6sE2LHd3JaM2UmJafbhhkuWKttSVMOBBxzNbCWAH/TsXQ43o+4JiteduKprQzWmtrgYjdZqzm8sSa23l09HzTNBKFVBLr9O32xmvJJmvJLVnL93TQ9o2kNfqmpG60pqL2bo3W9mWNfts9ul6j3916fm/mYNHzIdYX0yUptd2hlLKAm4FZwErgRaXUchF5QETOSzr1CuB5pVSy9DMK+EREFgPvAr9IXvWzt2h5R6PRaDrQnU6lSqmZwMwOx+7rsP+jTq6bBxzRbR1x0IO+RqPRJCHOarBDFT3oazQaTQf2IJDb69CDvkaj0XTgUB70e0Ugd8PG7fzg57fz3shJTLnyKgp+fj0bWiOc8PEsbv7h3ymfcg5PTLD400OzOX+AD++9T/CTR97Ak+njuRdXMMaXzjFP3MdtM75g1ew3yOk3nBsvP5Lhm2bz8W/eYUNrhKkFXioe+TXvLdkBwPHD8hl2/VdZIX15+p11bF/+KXY4QE6/4Qw5spTV/hCmwPAsD4OmDaD4lJNpKh7N+xvq+GBZFdXrt9BaU4mK2gSKR7CkqoUP11SzY0sTTds2EGysIWqFMVwe0rLzyCgoI7c4k4F9sqlxDNRsFUuw8ppCjsugKM0ksySD7L5ZZJUVkVlWRGOksyBu7Jp0JwCc5RKy0lwEWyOEHKO1eBDXDgWww0HsDtWqAJQ7g4i4CFlRgk7VrGAkSmukLTEraEcJhG0nEWtno7V48NZ0GRhmLEHLtlQsgNuF0RrQrh+dGa0lqmkJicSs+FfyzoKqyYlZu6pulWy01vFYV+xJELcnA5b7q2LWHqxh751I7D2m0nojeqav0Wg0SQixycmhih70NRqNJhk5tK2V9aCv0Wg0HejNxeV3R6/4DuPOyObGlU/y5pYmZp8Jjzz1GXc/8TWm/vYzQo01vPGjU5l57LWYIpw281Eu+P1H1KxeyPGXn4vfinLhD07lLe84pr/wPipqc9SZx/Gdw7JY/OPf8/aOFvp73Uy69mg+fG4plY7R2pgbTiAw4Ss8OqeCis++oKV6M968UvofPpqvTxlAwFb097oZdUQxA86cBEecxCfbWnh9yTa2ra/DX7UBK+jHcHlYWx9ibkUtqyvqqd+6vVOjtZxCH0UlWRzZP3cno7Ucl5kwWsvuk0VmaS5ZZUW4isqcxKz2Rmvx4ilxozWf2yQtJ41wwIolZiUZrdnh4E5Ga3HiRmvBJKO1eEJW3GgtEI61hJ7fwWjNcBkJo7V4IZWujNaS+5AwfeuQnJVI0jKNhI7vNoyYtt/hDzWemNWVnr87ozVDuleDP1iN1g40B1vXY4ZrqbXeSI93W0RMEflcRF539geJyHwRWSMiLzipyRqNRnNwEF8ckELrjeyPz6pbiaUfx3kIeEQpNQyoB67bD33QaDSaFBEM00ip9UZ6tNci0g84G/izsy/AScDLzinPAN3iEa3RaDTdgeiZ/j7xW+AuIOrsFwANjgkR7LqgwA1O8YBPStKC3H/bK9z/+BU8PPEGvja5jGdHXsvi/zzPrfdchzxwHa9va+Zb953OL7b1ZdFrLzP4+PN58apxXD5tIK7vPMSdf15IXcViBk89g8cuOZLqR+9l5rsbMQVOmdqP/jd+l4X1Afp73Uz+yghyLrmR55bt4MO5G6nfsAzT46Vo5AROm1zOmUPzyfeYjC3NYtAZR5B+zLmsDaYzc0UV61bX0rDpC4KN1Yhh4s0r4aPNDXy8poaayianGHodEDNaS88rIbu4D/klmRxe5mNEUdZORmtFaSZFGW4yizPJ7ucju7yEtNJSXKXlO63Rj2v5mWbMZM3nNvFkuEnL8RAKRAgHAlgBP5Gg3zFaC+9ktBYntj4/ZrIWshTNoZ2N1vxBi9awvUujNdPlPDoaf6pGa/Ei7akYrRlGe8O1rozWktmd0Vr8cMd1+8nsq9Fad2jx+1PP7+616Qebnh+nO2vkHmz02KAvIucAO5RSnyYf7uTUTgsKKKWeVEpNUEpNKCwo6JE+ajQaTUdE6DwZsJPWG+nJJZtTgfNE5CwgHcghNvPPFRGXM9vfVUEBjUajOSD01gE9FXpspq+Uukcp1U8pNZBY4YDZSqmvEfOFvtg57WrgtZ7qg0aj0ewpQmqz/N76wXAgkrO+DzwvIj8FPgf+cgD6oNFoNJ0iAh5tw7BvKKXeA95ztiuAiXtyfc2yVVwyfhyPDbkGDy8z7L9vcta593PEOZdyr/cz7n5iIV+bXEbdNQ/y8HW/J6t0IE/ffixbv3cVE/78W8795yLWvv86BUPHc/81R9Fv4T946fdzqAxanNsvh7H3XMs8ux8eQzhxfClDb/o2c/3ZPD3rM7Yt/Qg7HKBw+NGMmVDGleP7UVi1iMNz0hhy2hCKTjuL6tyhvLWsinlLt1Ozfj2ttTGjtbTsfLJKBvH2iiq2b2ygaVsFwcaaWHDS4yXdV0hmUTl5JVmM6J/LEWU+huZnMNMxWstyGeS5TYrSTLL7ZpHTL1YtK7NvMa6ScvAV7xTE9RhtRms+t4HXY5Kel443L51QINJWLSsSjhmtRWLB3M4CubFKWVFC1s7VslqSErMCEbtdENd0xQzW4kZrIpJI0jJNYyejtagVRtntg7jQZrrWLinLZSSCt+6kBK1kA6zkIO7eGK0lT+D2xmgtcW0nRmvdHcRN9fW7535fkiCuxEz+DlW0DYNGo9EkIRzamr4e9DUajSYZ6b16fSocusKVRqPR7AWxmb6RUkvpfiJniMgqEVkrInd38vw1IlItIouc9s2k5652LGvWiMjV3fH+esVM31Yw4L9vcubZd+Nf8CRD7nidzKL+zPv+FJ7oO5FR2WlMeuNVjrh3Nq21ldz1wC2M/fSv/PIvn5FzWRbzXn4JT6aPS796Ahfl7OD9u//K3NoAY3zpTP7+6VSPvZD7nv2MO4qzGHf7+WzuP5WHXl7K+k8+I9hYTXafIQweP5JvHjOQ4UYtNdNfZMTkMvqfezLW6JOYs6ae6Z9uZVvFDvxVG7DDAVzpWWSVDKSwvISKdXU0Vm6ltbYSOxxADBNPpo/MonJyizIp65vNkf19jCzMpDQz9l+SrOf7ijPJ6ZdNTnkx2eUlmCXlGMXl2NklOxmteZ2krCyXQY7bxOvo+el56VgBP3Y44Dx2XjglmZClYoZrVjRJz48SsmKFU+KJWfEiKobL01Y0JW62ZkpC3zcMwXQJthUlakexLStROKWzxKw47QzXRHAbbTp+3GjNlJ2/ku9Kz4/FCtobrXVVOKWjzr+nHIjCKQe7nn+w010zfRExgceAU4kloy4UkelKqRUdTn1BKXVzh2vzgfuBCcTymT51rq3flz7pmb5Go9EkYUhbBvjuWgpMBNYqpSqUUmHgeeD8FLtyOvCWUqrOGejfAs7YqzeVhB70NRqNpgOxFWK7b0Bh3C7GaTd0uFUZsDlpvyvrmYtEZImIvCwi/ffw2j2iV8g7Go1Gs7+QTqTCXVCjlJqwq9t1cqyj9cwM4DmlVEhEvk3MiPKkFK/dY3rFTL/0sMFM+dbTlB11MifPEqqWzmHGI1cx95hTqQxGuOb1Bzj9b1+w/sPpTLr8Uu4f5ufFG/5CY8TmN4+/Q6C+inHnnslDpw9m2V33MHNlDaXpLk698kiyrvkhP5u9jpUffM5RtxwPZ93Mbz/YwJIPVtK0ZTXpviL6HTmOr584mGnlWYRn/5M1//mMYRdOwZh4Lgu3tfKfRVvZtKqGxk0rCDXXYbg8ZBT2JbdfOYOH5FO7tQZ/1QYiLY1ArHBKRkFffCWFFJflMH5AHqOLsuiX4yErVOcUQo/p+QV56WT1zSK7Xx7Z5SV4ygbg7juQaFYhIU82kKznC5lmbH2+z22Q5vOQ7uj56XmZWEE/kUCb0VpnhVPiiGEStGMF0f1hC7+zHr81YuMPWW16fsQmELYw3B5MlytmORtfk++Sduv1DTNmWZtcOGVXRmtxvT9huJa0Lr+j0Vp8nX5nhVO6IpXCKXtqtJZ8n47X7y+jtd6w8ORgDxF0Y0buFqB/0v5O1jNKqVqlVMjZfQo4KtVr94ZeMehrNBrN/iKenJVKS4GFwDCneJSHmCXN9PavJ32Sds+jrf7ILOA0EckTkTzgNOfYPqHlHY1Go0lCkG6zYVBKWSJyM7HB2gSeVkotF5EHgE+UUtOB/xGR8wALqAOuca6tE5GfEPvgAHhAKVW3r33Sg75Go9EksYea/m5RSs0EZnY4dl/S9j3APV1c+zTwdLd1Bj3oazQaTTsOdRuGXqHpr6iOEGquY+nDZzHv2We464FbyH3wel5cuoPv/vRsftE6ho/++S8GH38+//32RN674EY+rgtwxSmDqP7iY4ZNO5+/Xn0UNQ/dxmsz1mArxVknlDPo+/fy1NI6Zs5cTl3FYgqvu5OnF21j5ttrqVm9ENPjpXj0JM49YRBfGVmIzHuR1S/MYcmyajJPuoi1Vg4vL65k6bId1K1fQaC+KlEtK7f/cPoOymPaqGKaK9e2q5blLehLTmk/CvpkMX5AHkf0yWFQbjr5RghX3UZ8TlJTKqq+AAAgAElEQVRWUYabnH7Z+MpzyRnYh/T+/XH3HYidU4qdVUR9MBZM7KxaVkZOGum5TmJWrpe03GwiwVhyVtxobVdBXDHMTqtltYR3DuImKmeZyUZrXSRpuYx21bKSg8mdBXGTDdeSq2XFE7Tc8eNOQLczOkvM2uk976JaliHtl1HsLoibfM84XQVx93Zs0dWyehBdREWj0Wi+PMT99A9V9KCv0Wg0HdCDvkaj0XxJMA7xIiq94p0Fmxp488+38d7ISUy58iruqnuZ3/7pE7594QiWfuU+fvPgM+QPHsNr/zuNNd+4iBeX7uCCwXmMf/qPlI6Zxm+/NYmStx5lxqMfUBm0OGtYPuN+chtv+Iv540vLqFo6B3emjzdq0vnzjJVULp6DitoUDj+aY48dyNVH9SN/w1w2vDiDZR9uZrU/xNbsIUxfWcXcRZVUrVlFS/XmROGUnH4jKB2Qx0mHlTClXx6B+qpE4RRvXgnZJQMoLMvhiIH5jOnnY0RhBn0yDFy1GwhXLKfQY1Ka7orp+f1yyBnUh8zyMtx9BqLy+hLNLqY+FKUhaCcKp7Tp+QaZXlc7ozVvgY/0ghzsUICoFdll4ZS4ni+GmdDxm8NthVP8QStmthay8AcjCcO1uF7vcpttBmtJhVMSWr8hXRZO6ajnx/G4DNyG0WXhFNNoX0Rld0Zrife6i8IpyXp+V9enii6c0sZBr+eD1vQ1Go3my4SQ8NU5JNGDvkaj0XTgULaS1oO+RqPRJCHQ5fLfQ4FeMej361+KcdOlvLmlidlnwg/GPs15Q/PJf+oVzrj+L4hh8Ni9F5D52B088dJKJud7OeXlB/nR4ij/+53jOX7Hu/zf7c+xuDHIKcWZHPvQ1Szvezw//vMCNiyYjRgm5UdP46HpK9iwYB6RlkbyB49h9JRh3HLcYAa3rGHr88+xasZqljWFCNiKN9bUMmP+Zrat3oh/+waiVhh3po+csuGUDizimNHFHDswnxEFaUStMIbLQ7qvkKzSQeT3yWZYeS7jB+RyWHEWZVlu3LVrsdYvo2XtmpieX5ZN7kAfOYNKyRnYB1ffQUhhP6zsEhotg/qgzbbmUMJkLa7n+9JdCZO1jMIM0h09P73AF1ujv4vCKcl6vhgm/rCNP2y1Ga0FY2v0m0NWYn1+IGxjReyYlp9srBbX8JPW7LscD/K4nh/dTRGXRGH0JHM1QwS3Ke0KpyRvp6rnw856fmfmaxAbBAyRPdLzO5sodtTzu3uN/sGu5/canN+1Q5VeMehrNBrN/kIAd4qlEHsjetDXaDSaJLS8o9FoNF8mnCXBhyp60NdoNJok4jGcQ5VeIVzlNm7jqdfXcP/jV/DwxBsYlZ3GiZ+9y7QfzKKpch3/e+81nLr4Kf700Gz6pru57K/f4Vl7NH964nWuL6zi/W8+xKyqFsbnpnPKT85nx9RvcOvzi1j9/ntYAT+lY6Zx5Tkj+WLOR7TWVpLdZwjDJh/Jd08exhhXNTUv/ZUVLy5iYX2AxkiUojST5z/ayOYvttKweSVW0I8rPYucPkMoGVzG+NHFTBtWyOHFGXh3rEIMMxbELRlEYVk+gwfkMmlwPmNKcuif7Sa9YRP2ppUE1n5Bw5rN5JdkkjsgB9/AEnxDynD3G4pZOgjb14cWSac+ZFPZHGJrcxBvUhA3zxNLysoozCCjwEt6QXYiiOvKzcd2qmXFA6idEQ/imm4PzaFYxaxmx2QtYbQWthOP4bCNbUV3NlaLJ2S5YtWyXEnFpDsmZXVltBZvbeZqRqJKVnKiVlvlrLb3karJWvJ2PIjbLri7b7+6Xf6BdX/Qtbvv1/2DXm8aR2PGfrtvvRE909doNJokxJlQHKroQV+j0WiSONTlHT3oazQaTQd6q3STCr3iO8y27c18/47jeGzINQBcufhlJv50LlsW/pcrb/8G/2PP4w83/B1ThOsfvpj3hl/GfY+8ReOmlXx09R38Z1Utw7M8nHvXydhX/JCbX1nK0rfmEKjfTvHoqZx35ghumtSP5m3ryCzqz9DJE7ntjBFMK7Jo/s9fWP6P+SyobKY6ZONzG4zPTWf9sm00bFhGpKUR0+Mlq3QgxUMGc8SoYk4bWcy4Pln4GtcTXjaXdF8RWSWDKOhfTP8BuRwzrJBxfXIYmOshs6UKtXklwdXLqF+9mfo11eQNzsU3qBjf0DI8/Qbj6jsY21dKqyuL2oDN9uYw25pDbKkPkOMyyPeY5HvMmLlaoZeMQi/ewmzSC3xkFOfhzsvDyCnYpZ6fnJRluj2J5Kx2SVlBC3/IojkYSej5VsTGikQxXAYud3vTNZfbSBRWiev5aS6jzXBtN3p+nGQ932UaSRp+m57vNtv8UlLR8xP3lt3r+YbIXunR3V04pcvX6QUDVG+aOAttBn67ayndT+QMEVklImtF5O5Onv+uiKwQkSUi8o6IDEh6zhaRRU6b3vHavUHP9DUajSaZbqyRKyIm8BhwKrAFWCgi05VSK5JO+xyYoJRqFZHvAL8ELnOeCyilxnZLZxx6xUxfo9Fo9hcxTT+1lgITgbVKqQqlVBh4Hjg/+QSl1LtKqVZn92OgXze+nZ3Qg75Go9EkEbdhSKUBhSLySVK7ocPtyoDNSftbnGNdcR3wRtJ+unPfj0Xkgu54f71C3inOS+fDrz7IT7/zIP4FT3Lss9tZOetlTv/O9Tw+YgdPnfBz6iM2t/34TNaccSffeeBNdqyYy5ATL+CF391K33Q3F944Bd9tv+Fbryzjoxnv46/aQOHwoznt7DHcc9Jg0mb/GW9eKYMnTeHGs0dyzoB0gq/8lqV/m8NHa+upDFpkuWJ6/rCTB1JfsZhgYzWGy+Po+cMZObKQMw4r4ei+2RS2VmItm0vNx5+RWTSBvLJS+pbnMnVYIeP7+Bicm0ZOsAbZsoLg2iXUfbGRulXbqV/fwJAzRpA3vD/pA4bgLh+O5etLIC2PmlaL7f4wW5uCbKpvZWNtK8e4Y2v0M/JjWn5GYQbegiy8RXkxPT83FyO3GDOvaLeF0A2XBzHj2278YcspltKm5wfCsSIqgaCV0POtiB0zVUtan2+YktDzvR4zoed7XGZK6/MhbrgW7VTPdydtx4qiyx6Zoqmo3a4QOnSt5+8Nqer5+5wH0ANa+aG8ciUlBPZgxWaNUmrCru+2E6rTE0WuBCYAJyQdLldKVYrIYGC2iCxVSq1LuXed0GMzfRFJF5EFIrJYRJaLyI+d44NEZL6IrBGRF0TE01N90Gg0mj0lvmSzmwK5W4D+Sfv9gMqdXlPkFOB/gfOUUqH4caVUpfNYAbwHjNvrN+bQk/JOCDhJKTUGGAucISKTgYeAR5RSw4B6Yl9nNBqN5iBBHDvv3bcUWAgMcya7HuByoN0qHBEZB/yJ2IC/I+l4noikOduFwFQgOQC8V/TYoK9i+J1dt9MUcBLwsnP8GaBbdCqNRqPpDrpzpq+UsoCbgVnASuBFpdRyEXlARM5zTvsVkAW81GFp5ijgExFZDLwL/KLDqp+9okc1fWe50qfAUGLLltYBDc4PAnYR1HACIjcA9MlI78luajQaTYKYDUP3xTWUUjOBmR2O3Ze0fUoX180Djui2jjj06OodpZTtrDHtR2zp0qjOTuvi2ieVUhOUUhMyBw3n2//zG8qOOpmTZwmfvvRPpl59Da+dbPKPk25ltT/ETXecQN01D3LFL96l8tNZDDjmXJ68ZSr5HpPLvjGOPvf+jjv+bxWzXplD05bV5A8ew4lnT+D+04aR+9E/+eyXLzFo8rHccM4oLh+Zi/V/j7P0L+8yb1k1mwMRslwGY3xpjDxxAIMvnEagfntSEHckI0YXcf6YvhzT30dJuApr6RxqPlpI5fwK8vv3p+/AWBD3qDIfQ/PTyY3UI1tWEFr9OXXL1lO/ahv1FQ1U1wTIG15O+sBYENf29SWUUUBtwGJHS5jNjQE2NQTYWNvKlrpW8j0mWXnpZBR6ySzJJLM4G29RHt4CH56CfMy8YkxfAUZ2PlErvNPPuWMQ13R5MFxuDJcHf8iisTXSLojbHLQIJSVlWRGbqBVNVM5yeUwMUxIJWslJWR6X2VY5K8UgLpComtVVENcdr57VyW9zVxW5oC2IG6+glfiZOI/xmdy+xDUPZBB3b+7/pQ/iOoik1noj+2X1jlKqQUTeAyYDuSLicmb7nQY1NBqN5kBi7PNH8sFLT67eKRKRXGfbC5xCTNN6F7jYOe1q4LWe6oNGo9HsKYKe6e8tfYBnHF3fIBbAeF1EVgDPi8hPiaUf/6UH+6DRaDR7TG/wM9pbemzQV0otoZM1pc5604l7cq+KDdsp/+pUlj58Fr4pNzLlyqt4+4Ic/jnhChY3Bvmf246l9bZHufCns9n00euUTzmHP91+LBNWPE/pNWPp/+CT3DFrI68+9x4NG5aRO/BwTjh3Cg+ePYriT17gswf/wTufbuNbvx7N1UcUYs/4HYsem8W8RVVsaI3gNYUxvjSOOKGcYZdMw33cxRiuX5BVOpCioaMZNrqIC8aWMbU8lz6RaqLL5lAz92Mq56+jalk1pefncvyIIqYMyGNUYQYFVj3G1hWEV39O7ZJ11K7cSu2aenbsaGF70MI7ZBiegSOx8/oTyixKJGVtagyyqSFARXULG2taaGoIkpWXTmZxZkzTd/T8jOI80ooLY3p+XjGGr5Co17fTz3VXer7h9rTT8/3BSELPD4csrEiUqBVrViRKeoa7Uz3f6zHb6fke09gjPV9FbcdwTXar53fUo3el58dJ1vMN6VrP35uvxFrP76X04ll8KqQ86IvIMcDA5GuUUs/2QJ80Go3mgCGkvAa/V5LSoC8ifweGAIuA+FRJAXrQ12g0hxxa3on5QYxWSnW6vFKj0WgOJQ7hMT/lQX8ZUAps68G+aDQazQFHl0uMUQisEJEFxDx1AFBKndf1Jd2Hy5vFykfP5t2Rk5hyy6O8+5Usnj0qFsS99Y7jab3995z/wDtsnDeD8inn8Jc7jmfi8n8x/ZtPcMG6edzmBHHrKhaTP3gMJ5w7hV+dNzoWxP3ZM7y9oJLKoMVdRxYRnfE7Pv/9TD74fHsiiDs+N50jTxrIsMtOxn38pXxh+RJB3NFHlHDB2DKOH5BLX6ua6NL3qP5gHlvnraVqWTWrmsNMG1W8UxA3tGJBp0HcmrCdCOIGM4uodoK4G+oDOwVxW5tCZBZntkvK6iqI2zGQu6sgrpnmxXR5dhvEjSdo2VY05SCux2XsURAXSDmIm6zD6iCuZl84hMf8lAf9H/VkJzQajeZg4lAuNJLSoK+Uel9ESoCjnUMLkt3gNBqN5lBBurFc4sFISh9oInIpsAC4BLgUmC8iF+/6Ko1Go+md6IzcmLn/0fHZvYgUAW/TZpHcoxzeL5s3Bk1gTk0rs8+EPx91Jav9Yb5372lUf/MhvnLfm2xdOJPBx5/P3+84nsM+eoKXb3qWubUB3phRwf+98A6Nm1ZSMHQ8p3/lGH525gjy5/6NhT/7F+8sqmJ70GJghpvIy7/i88fe5IOlbSZr43PTOeKUgQy97FTM4y9jZTCTFxdXUjLsMA4/soSvjCvj2P4+SkPbsBa/S/WH89kyby3bV9Sw1h+hKmRx7sB8RhR6KQjXxiplrVhAzZJ11K6opG5tHdU1AbYGLOojNn4ripU/gFBGAdWtFlubYiZr6+tilbKS9fzW5lA7PT+zT0GbyVpBKZJTSDQ9O6bpp2Unfp6p6Plxw7XO9Py4yVpcz7ftaMp6fprL2CM9P1bhKjU9P67Fp6Lnw64rZXXU82Uv/8J3p+d394Syl45DBxWClncAjA5yTi2H9s9Fo9F8idnbD/neQKqD/n9FZBbwnLN/GR38oTUajeaQQHRyFkqpO0XkImLlugR4Uin1ao/2TKPRaA4AQqyGw6FKyt47SqlXgFd6sC9dUrd0FR8bfbj/8St4eOINNFk29zxyEZ+ffhfX3v0fqpbNYdTpF/PC7cdS+tov+Mf3/81nDUGmFWXwnWdfx1+1geLRU7ngwqN54LShpP/3D3z881eYvbKG6pDNkEwPx08pY+GvZ/LhmjoqgxY+t8HReV4OO2sIgy47B2PKhSxuNHnu8818sKiS8eP7cIFTNKWoZRORz2dT9cECKj9eT+UXtaz1h6kKWQRsxeiiDHIDVbBpKYGVnyX0/Nq19eyoC7A9aCf0/HBU0ZqeT02LxdamEJsag6yvbUno+f6GIC1NIQL+EKHmJrL6+JL0/AKM3GLMvKKYnu/1obw+7LQsWiMxrTxZzzfdHmfbjenxYrg9mC5PbNvloaE1TCBsEwha7YqmWGGbqK2wnbX6dryIiqPlJxdN8XpMPGZ8P9b2RM8H2un5brNNv++o55tG6no+dK7nd5eWn3z/OFrP7z0cyvLOLnV5EfnQeWwWkaak1iwiTfunixqNRrP/iGXkptZSup/IGSKySkTWisjdnTyfJiIvOM/PF5GBSc/d4xxfJSKnd8f72+VMXyl1rPOYvavzNBqN5lCiu+b5Tj2Rx4BTidUEXygi0zsUOL8OqFdKDRWRy4GHgMtEZDRwOXAY0Bd4W0SGK6U6/+qaIqmu0/97Ksc0Go2m9xOTC1NpKTARWKuUqlBKhYHngfM7nHM+8Iyz/TJwssT0pfOB55VSIaXUemAte1iLpDNSXXZ5WPKOiLiAo/b1xTUajeagI8XELGfMLxSRT5LaDR3uVgZsTtrf4hzr9ByndngjUJDitXvMLuUdEbkH+AHgTdLwBQgDT+7ri6dKOKr48et38xv3iXh4mR+8eCvP9b2Au+/6O01bVnPUJV/j1ZsmY//2uzz1q3dZ1xLm3H45nPTYN7nqx0spO/osrr3kCO46tpzg33/CnIfe4O1NjfitKIfnpDF12gBGfftifnbBQ1SHbIrSTCblZzDywlH0v/h81MQL+LCylec/28iCRZVUranggcsuYWJZNrm1qwl9+jbb5nzC1o83saWigbX+MDVhm3BUYQrk+TcTXb+Y1uWLqF1eQc2KKuorGtjeEGR7sC0py3aMq6taY0HcDQ0BNta2UlHtp7IukAjiBlvDhJqbiLQ2kjG6gMzSAtwFcZO1IsgqSJis2e4MWsJRWiPRtoQsw8R0xxKwkitluZwAbjxJyx+0CIftToO4VtjGtmPJWVE7iscJ4HpcBhkes11SVnIQ1+My2gVx24K2bUHc5MBrNGqnHMTtbObVVRA3TqpB3D0Nuuogbu9FlEJ283uTRI1SasKubtfJsY4W9V2dk8q1e8wuZ/pKqQcdPf9XSqkcp2UrpQqUUvfs64trNBrNwYioaEotBbYA/ZP2+wGVXZ3jqCg+oC7Fa/eY3a3eGelsviQi4zu2fX1xjUajOfhQoKKptd2zEBgmIoNExEMsMDu9wznTgaud7YuB2U7BqunA5c7qnkHAMGIeaPvE7tbpfxe4AfhNJ88p4KR97YBGo9EcdHRTkUCllCUiNwOzABN4Wim1XEQeAD5RSk0H/gL8XUTWEpvhX+5cu1xEXgRWABZw076u3IHdL9m8wXmctq8vtC/0OWwQX6scw8w/PYp/wZPcuaaQv9z1BFErzBnfuoYXLh9FxS1X8K/nluO3onx1Yl+m/O4uFpYcx9AT5nHX18by1XLFjl/exkePf8icmlYAphZ4OfqCEQy5/hoaRp1Gdejn9Pe6mTggh5EXjaXPRZfQMvwEZlc08Pwnm1m2pIod677Av30DJwzwkb75U1o+foutcxZRuaCS9Vua2BywqEvS831uE/uL+TQvW0ztsvXUrqqhvqKBrf4w1aFYUlbAbtPzPYZQURdgU2OQDTUtVFT7qaoP0NIUorUxRKs/RKSlkXBrI1bAT1ZZEe7CEgynaAqZuUTTfUTTc4iYabSGbVoiUVojqks9P9lkzUyL6fouj5tQyMIKx4ul2E4yVqyASrKeb1tWkpZv7FLP9yQbriXp+R0TsqJJmqrbNDCE3er5HSX9VPT83Rms7av23tnlWs8/yFEq1Vl8irdTM+lgW6OUui9pO0jMwbiza38G/KzbOkPqSzYvEZFsZ/uHIvJvERnXnR3RaDSag4Vu1PQPOlJdsnmvUqpZRI4FTie2pvSJnuuWRqPRHCgURK3UWi8k1UE//j35bOCPSqnXAE/PdEmj0WgOIIruDOQedKRquLZVRP4EnAI8JCJp7Ec//ZW1Nkt+/yfKp5zDybOEj//1B7JKB3L7bRdy92A/8049i5cWVJLvMfnGJaMY/cuH+Fd1Hr/43Twev2kKx1HB6nt+zrsvf8HixiA+t8FxhZmM+cZEyq79FhU5o/jb3I2Myk5jwpHFjLh0Innnfo1tvuH8d/kOnl+wmQ0rdlBXsYzW2kqiVhjPineomzubrR+uYNun21lb00pl0KIxYmOrmDbvcxv0TXdTN38+tcs2ULe2ntqNjWwNxAqgN0Zi2n+ynp/lMlhd20LFjhY21rZQWx+gtSkUW5/fEiTcXEck6McK+LHDQdzFQzALSjHzihPFUpTXRxAXreEoLZEoAStKc8hKGKolr82P6ffe9nq+Y54WCdmJ9fgxTV8lCqjENX0VtYla4YSe7/W42hVMiev4piE7afqwez1f2XY7Pb/jWn1o0/ONJHV7d3p+/DpITc/fG/+tnl6b39lraLoDBdHeOaCnQqoD96XEos9nKKUagHzgzh7rlUaj0RxADmVNP1U//VYRWQec7ji9faCUerNnu6bRaDQHiF46oKdCqqt3bgX+CRQ77R8icktPdkyj0WgOCEpB1E6t9UJS1fSvAyYppVoAROQh4CPg9z3VMY1GozlQ9FbpJhVSHfSFthU8ONv7LYYUaKxn6u1f581bpuCbciPlU87hye8ex5QvXuTVyY/z9o4WxvjSOf9708j73iPcOWstL738NlXL5jD5rBrm//xvvPXRViqDFn3TXZx4ZDFjbjiJjPNuYG5zJk/OWsWC+Vt44dSBDL/8ZNwnXMoXdj6vfLqVNxZuYevqSho3rSRQvx2AtOx8ql6fztaP1rB98Q5WNceqZPmt2C+K1xQKPS7KvC76FHjZPn8NtWvq2bGjJWGw1hiJVcmCWGm2eBA3x2WyfGsTG2taaGoI0toUorU5RKjFT6SlsV0Q1woFcJWUY/gKEwZr0bRsWi1Fa8SmxYoSiERpDFo0hqydgriJAK5TNcvlScMwDVweE5fbxEoyW7Od4G27xCwrHAvkRsKxAK6TkNUxiJtopoEhsssqWfEgrrKTkrMMY5cJWfEArkhqAdzk19tdEHdvCyilEsTdl+pMOoDbk3RvctbBRqqD/l+B+SISr4t7AbHUYY1Gozn0+LIP+kqph0XkPeBYYpOMa5VSn/dkxzQajeaA0M02DAcbu/PTTwe+DQwFlgKPOyb/Go1Gc0gifLk1/WeACPABcCYwCritpzvVkb79Snn3tAhvjZzEMbf+jv9cfzT1P/oWDz/+MZXBCF8Zls8Jf7iJiiMv4at/nM+SWe/jr9qAr3wUb1/7CO9u9xOwo4zPTWfKGYMZfv3lhCdfwr9W1vD0e0tZ99k66jcuY/SvbsAedzazNzbx4ufr+GTRNqrWrKGpch1W0I/h8pDuKySn3wjWzHiczRsaWN8SaVcwJctlUJIW0/OLy7LJH5ZP5YJKKptCbA/aNFntC6aYAl7TIMtlkOc2yfcYzKxswt8YJNAcJuAPEWpuSBis2eEgVjhANBImaoWR/D7YXsdgzeWlNRwlYClaIlFawjaNIYvGYAR/2Mb0pO+s5ycZrJmmgcttYrgMXG6DcMjq0mAtruXHk7O8HrNLgzXTEDymgdsQDEN2WTAF2uv5KmrvVs9P6PIpCt3Jev6uCqYkS+77kol4qOn5+9D1XoICu3euzEmF3Q36o5VSRwCIyF/YAy9nEekPPAuUAlHgSaXUoyKSD7wADAQ2AJcqper3vOsajUbTA8RtGA5RdjeBicQ39kLWsYA7lFKjgMnATU5197uBd5RSw4B3nH2NRqM5aPgyZ+SO6VAbN14rVwCllMrp6kKl1DZgm7PdLCIriRX1PR840TntGeA94Pt7+wY0Go2me/kSB3KVUmZ3vIiIDATGAfOBEucDAaXUNhEp7uKaG4hV7aLMl8VDU26iJmzxzumK9485kVeW7aAkzcUt3xjLkJ/+hqc3unj4Z7PZtOAtAMqnnMMlZ49kxjmPke8xOaU8jyOvnUzJld9ijXcwT761jrfmbqRy2SL8VRtQUZttw09n5qLtvPjxJjZ9UU1dxRJaaytRURtXehaZxf3JLx9G6cBclr1ax+ZAJKHPewwh32NSkuaiPNtD3uBcCkYUkje8Px/MqkgYrAXstoo8bWvzDXyOnu/LTqOhuoWAP5wwWAu3NmKHAljBFmxHy4/r4XZ2ESotm4AyCSQZrDUELPzh2Pp8f8iiKWQl6fedG6y53CYuj5HQ9luaQjutzY9r+HE9P6Hpu82d9Py4yZrbMDAlVgzFbchuDdYS285xt2HsVPw8Wc83JDWdueMa/lQM1g4mLX/PX797X+vQ1/KTOIQH/R53yhSRLOAV4DalVNPuzo+jlHpSKTVBKTWhINPbcx3UaDSaZA5xG4YeHfRFxE1swP+nUurfzuEqEenjPN8H2NGTfdBoNJo9Q6GsSEptXxCRfBF5S0TWOI95nZwzVkQ+EpHlIrJERC5Leu5vIrJeRBY5bWwqr9tjg77Evsf+BViplHo46ankyu9XA6/1VB80Go1mj1Hsr5l+KotaWoGrlFKHAWcAvxWR3KTn71RKjXXaolReNFUbhr1hKvB1YKmIxDvzA+AXwIsich2wiS4KAms0Gs2BQKHaxZZ6kN0ualFKrU7arhSRHUAR0LC3L9pjg75S6kO6ziM5eU/uVVnZSFFuPjf99hIenngD61rCnNsvh5N+fy1bp36TM19YzKJZH9K0ZTXZfYYwetox/PC8wzg5p5EnctKYOm0Ao759MerEq3hpVS1/+s8i1i3aSO3az4i0NOJKzzbfhgYAACAASURBVMLXbzi/eHcdH39eyfbV62jato5ISyNimGQU9CW7z1CKB5YydEg+J44sZqU/nEjI8rmNhMFaaZ8s8oflkT+8L3mjBpA2aCSbAzO6TMjKcRnke0wK01xkFHrJLMmkuT5AqLmJSGsj4ZbGnRKykgOSljeflkiU1kSFLJvmsEVj0MIftmkMRfAHLRpbI7jTs2LJWU4Qt7OErHhA13QZTrWsrhOyEoHcqJ2onNVVQlY8mOsyjZQSspLZXUJWx6pZndGZEdueJGTtaQD2QAZxuzuAC1+2IC57UjmrUEQ++f/2zj06jrvK859b1d1SS7L1lixbduT4HRISEscheGFCEkiGJY/NJiGBYZhdMh4WGOAAQxIyBJiznA3MbMJhYQHzZiYDA4EcAgRMEvJYHiE4iZ3YsR07fscvSZZkPVrqrq7f/lG/blXL3VLLD0ntvp9z6lTVr54/u3X719/7u/eG9tcaY9YWeW1Rk1oyiMgqgjK1r4SaPycid2N/KRhjRiZ66Okc6SuKopQgZjLSTZcxZmWhgyLyKEGA6ljumswbWf/nvwLvMSY7tehO4BDBF8Fagl8J/zTRvdToK4qihDHmpJ20o7cyVxY6JiKHRaTNjvILTmoRkdnAL4F/NMY8Hbr3Qbs5IiLfAT5ezDtNWXFzRVGU0sBkpcuJlpNkwkktIhIDHgS+b4z58ZhjmVmQQpDuflMxDy2JkX5zXSX/bcsv+cLmNDEe4BMfeQPtd9/LvRv6+canfsOrzz6CE4lx9puu493XruADl7RT+dT32PDlB3jHZ95G/c1reEnm8tVfbOOpP+zl4EvPM9i5D4Ca1g4aF53H2a9p4Ve/3krv7hdJ9BzG+Gmi1bXMau2grr2DtoX1rF7WzOqFDbympZqNviHuCvVRlzmVEebXVtCwpIGGxY3UrziLmsWLiXaswG88i77UqD4Yd4W4O6rlN8RcZtVWUNNSTVVTnJq22Qx1H85JsDY2ICtMz3A6q+dniqUMWE1/MBlo+b1DKQZGPNxYPCcgKxJzA00/FJAV1vazmn6oWEo4IMsPffjjIU3ftRp+1A2SpI3q+pLVmycKyArvR92Qhp8nICus8Y9loj/MU63l52O8exSbJK5YNCDrFJCZvXP6yTupRURWAu8zxtwG3Ay8CWgUkb+x1/2Nnalzv4g0E/hONxBkRJ6QkjD6iqIoU4eZjCP3xJ9iTDd5JrUYY9YDt9ntfwP+rcD1l5/Ic9XoK4qihDFM1ZTNaUGNvqIoSg6Tmr1TcpSE0ffaF3LhvVvZ8eTDDDyzlkfdc7jp3ud4+clHSQ720bz89ax+y3l89i+Xs6T7OXbd8Y88+6NNPH00wcfvf4j7XjjIA08+w56NL9G3/2XSyQSVtc3UdZzL/OXzuPJ1c3n7ilbe9L1/J51M4MbiVDXOZXb7MuZ01HPB0ibeuKiRC+fOpmN2lOjhbaPJ1aoiNJ5VS9OyRuqWtlO3bCHRjuVI2yK8unZ60sE/ccwR4q5Q7Y5q+fXVUeJNVdS0VFHdWk28pZ7qOQ0kfnUoW/i8kJYvjos4Lr3Do/PyM3p+/0ig5Q8MB9sDwyn6hz2i1bWj8/CzBdAdq+OP0fcjDl4yFTw/nTsv3/hp0pl9OyLKaPoZDT9qi6BH3UDHjzqCazX9Yubmh/fHJlcb2wbHa+PFONnG6vnjafknqr0X0vNVy5/BnMLZOzORkjD6iqIoU4eO9BVFUcqHqZu9My2o0VcURQlhMNk6zmciavQVRVHC6Eh/+nll9yGij/+c9ovfyhXrhBfWfY2Bw7upXbCClTdcy2euOYfVFYc5/LV/4Nff/CO/PzLI0WSaOZUR3vntP7Nzw26O7tpIarCPaHUt9R3nMnf52ay+YC7XnDuHlW3VzD7yEsZPZx24LWe1sHRRA3+xrIVL2mtZVF9BvGc3/vqNHNu0gfNrK2iZNysIyLLJ1WIdy3HnLSVd384xp4rOQY/9x4aoiQTJ1eptdaz6WITq1iqqmqqobqmiqqWW6rZG4s31RJtbSf5kR97kahnEcXEiMcRxOTgwQr+tjNWfHHXg9iZSDAynGEqmGRj2SCbTxCoiOQFY2eCsqIsbERzXIRYKskqPJPImV8s6dNOh4Kyomze5WqZiliOS3S7WgZvBDVW2CidXy3HsMurMLDZSspiArHJz4EKZO3EhcOSmktP9FqeNkjD6iqIoU8fUBGdNF2r0FUVRxqLyjqIoSplgzKlIpjZjKQmjH6ms5vb/+RHu+IsOai99P7PaFrHqlndz13XncGXdAEe//1ke++bv+d3ePjpH0jRXuLy9bRbLb1jBPz/4s2yhlMbFFzJn6SIuPr+Na89r49L2WdQd3c7IukfY9dR6mpdfTdOCVpYuaeSy5S2smlfHovoYNf2v4j//PINbX6DrhR10vXSYZW+cn1MoxW0PtPw+t4bOhMerxwbZ3ZtgT/cQcysjxxVKyWj5QUBWI9HGJtz6Ftz6ZrzEhgm1fDcaFEM52D8SJFhLpOgbCoKwBkY8+odTWS3fS6XxUj6xePS4QimRqHOcll8RcYjHIqSTiQm1/Mx7VkSccbX8TKCWW0B3L/RHZvz0hFo+jBZYmewfa7Fa/snK3KrllxY6e0dRFKVcMAaTVqOvKIpSFhhj8FPedL/GaUONvqIoShiDjvSnm3Pnz+ZD27/FE3/3G97w4S9x9zXn8KZ4F0e+8xke/eYf+H8HBziaDLT8a9pns+LGc2m/8Xr8C6/BXH47TUsvpm3pQi618/IvnltDbddWRn79KLueXM+BP+9nzys9vPHej2Xn5S+sq6C6by/+888zsCXQ8ru3dXJ0+1EOHBvh5i/eQsWic7Lz8nudKjqHPPYfG2RvX4KdnYPs6R5kf9cQH5tVcZyWn52X39iE2zgHt74Fquvx47V5k6uN1fKdSBQnEmNvz1CQWG3Yoy+RzJmXnxoJ9Px02sdLpqmsjo47Lz8obm73Xee4Qin5tPyM9lnpOgXn5TsSzLXPFDgP9288LT9Dxg8wnpYPky8Dlzl/OrX8E7n/6dDzlVzU6CuKopQJxhh8zaevKIpSPpzJs3e0MLqiKEoYO3unmOVkEJEGEXlERLbbdX2B89IissEuD4XaF4rIn+z1/2GLqE+IGn1FUZQQmdk7xSwnyR3AY8aYJcBjdj8fCWPMBXa5NtT+eeA+e30P8N5iHloS8s7RF7Zy94d6iDnCY1cZdn3xA/zEVsZKpA0dVVGuXNbI8psvovWGd9A7fxU/3dnDD+/fyOuuuz5bGeu85kqiu/7EwI8eZduTGznw7CF2HuhnXyLF0WSau69alq2Mlfr9s/Rs3kT35l10b+2mZ2cv+4ZSdI54HPN84pffRLqunc50hM4hjz29/ezrS7DLOnAPdQ8xeGyEwWMjzLmgJacyVrylnkh9M059C5HGOfhVdfgVs/DjtSSd4Ms6UxlLHBcnGsOxztyMA9etiONGYuzvSWQrYw0Me0EgVtK3AVnWkesZfM+nenZlTmWseMylwjpxww7cTJuXTGSTo+Vz4I5uBwnXMpWxRqtk5TpwA+fu+EnR8galFUisNtaBWyjJWSEKOXDz3WWywVWnw4GrTB3+1DhyrwMus9vfA54Abi/mQgk+vJcD7wxd/xngqxNdqyN9RVGUMHbKZpHyTpOIrA8taybxpFZjzEEAu24pcF6lvffTInK9bWsEeo0xmZ8b+4F5xTy0JEb6iqIoU8bkInK7jDErCx0UkUeBOXkO3TWJN1pgjDkgImcDvxWRF4Fjec4zxdxMjb6iKEoIw6mbvWOMubLQMRE5LCJtxpiDItIGHClwjwN2vVNEngBeB/wEqBORiB3ttwMHinmnkjD6Sd9w04VtXLDmzdy7ag2vDCaJu8L5tZWc/8b5LLv1zUQvu4XtppHvbD7Eww89zb6tr9K3dwsbfnQn7X43/qaf0/Wd3/PqH7dzaOMRdgwkOTDsMeAF/7lxV1h86GmSTzzLgRd30LVpH93be+juHOTVhEdPKk1fyifpB1+m+yoXcLg7xe7eAXYfHWJn5yD7jw7R1zvM4LFhEv1JEv39pAb7aLtkMVUt9VQ0NWQDsZzaJvx4LV7lLPzKWoY8w1DSJ+F5VruPIa6LG9LxnWiMSCweaPqxOE40xp6uQUZCSdW80Hba80mnfXy7rqyOEsvR8kd1/EyitVho8VPJHN0+84cQbgPw/TSVERuQZbX8qOPk6PhhXb/YZGsZ3Ix2P4GWf7K6+9jLT3WSNNXxSwRj8JNTkobhIeA9wD12/bOxJ9gZPUPGmBERaQJWA18wxhgReRy4EfhhoevzoZq+oihKGAO+7xe1nCT3AG8Rke3AW+w+IrJSRL5pz1kBrBeRjcDjwD3GmJfssduBj4rIDgKN/1vFPLQkRvqKoihThWFqsmwaY7qBK/K0rwdus9t/AM4rcP1OYNVkn6tGX1EUJYwhp47zmUZJGP22c85i4bpH+MqGA8R4gFsuamPFzStpvuFdHGk+j5+80sMPHtzLK5s30vXKZgY79+F7SdxYnLoff45tv3uRg88dYsfBQQ4MB3Py0wZijtBc4dJaEWFBVZTtX/wyXVu76dndx6sJLzsnP5H2SVu/eMwR4q7wy+1d7DwSzMnv6kkw0DvM0ECS4cEkyf6jJIf68BIDpJPDNF5yUbZAil9Vh19ZS7pyFkknxmDKZ2jQI5Ey9I2k6B9JE43XZOfkuxVWww/p+G4sni2EcqxvJO+c/EyiNeMb0p6H7yVprIll5+THo26Oju86kqPnR50g4RocPycfAh0/g0mnqYg4eefkh/fDhVDC9xqPoIjK8UnV8un4J5KHrNg5+ZONAZjoGcpMxmgahhNBRL4tIkdEZFOoraiwY0VRlGljcvP0S47T6cj9LnD1mLZiw44VRVGmBWMM6aRX1FKKnDajb4x5Cjg6pvk6gnBh7Pp6FEVRZhTGSpoTL6XIVGv6OWHHIlIo7BgbzrwGYEFb6xS9nqIoZY9WzpoejDFrgbUA1fOWmkvWfIu+/S8z8Mxaeuev4pGdPfzwiX1s2/wYnTteYvDIPtLJBG4sTnXzfGa3L6N1QR0//uQHswnV0iYI9KmNZpy3ERrPqqVpWSN1S9v52b88XtB5WxMRql2HhphLQ8zl63/Yk02ols95640k8L0guCn6mr/Cr6wlZROqDaZ8hoZ9EqkU/UmPvmGPvhGPgaRH/4hHrKY+m1Atn/PWdR0iMZdI1GGgL5F13qbTQUCWn/azzluTTmffo6GmIieh2tjFtcnSMpWvfC8V/F8UcN5mt8PBWQWct+GkaRM5cMcezwRnjee8PZGfrGEH65nkvNXCWieJAZMuKqNBSTLVRr+osGNFUZTpwmCmKsvmtDDVEbmZsGOYRNiwoijKlGHA+KaopRQ5bSN9EfkBQa7oJhHZD3yaIMz4RyLyXmAvcNPper6iKMqJYAykkxqcNWmMMbcWOHRc2PFEJHp7iPUfZd5FV3DFOmHPlofp3f0iQ90HAs28upZZcxfRsGARczrquHRpM6vPbuS8lmr+16eGbRBWhLmVEebVxGhYUk/D4kYaVpxFzZLFxDqW4zd1sPFTv8o+M+4KcddhdsShNurSXOEyq7aCqsY4Na3V7N58gNRgX46On04ls/p5WJfur1/EYMqQSPgMpZI5Gv7AiMexEY++IVsIZcQjXj8nG5QVibpEYhkd3xZAibo4EYdI1OHI3r5RLd8+O5MozfiBnu/b7ZZZFaP6vQ3GijoOUVeyer7j2LVNjDaejh+mKurmJEQL6/ijursU1JvH0/lFZLSISuh6Z8w5k+W4hGvj3ONUJ19zTrHwrjr+KcQY1fQVRVHKCV+NvqIoSpmgUzYVRVHKBwP4JeqkLQY1+oqiKGGMUUfudDNnXis//caHOb+1itpL348bixOvb2XuRVfRuqCO1y5t4o2Lm7h43mw6ZkeJHt5G6uVf0P/rTVzVWk3jWbU0LK6nYcUC6pYtJNqxHGlbRLqunZ50hM4hjz09w9RGnZwArPrqKPGmKmpaqqhurSbeUk9Vcx1VbY30fGNjTgDWWEekOG522do9TN9w4LjtGwkCsPqGUgwMB9sDw9aJO+zhpdLUNDXlBGA5oaAsNyKBc9dWwNqzeX9OAFZmSWf206PZMZtnVxwXgBV1A6dt1MlUvRrdTqeS2f5MVO0q6jg5AVjhjJo57QWuHw/XemzHc9yeqKO1kPNWHbfli9HgLEVRlDJCjb6iKEo5oRG5iqIo5cMUReQWU19ERN4sIhtCy7CIXG+PfVdEdoWOXVDMc0tipN+S6KTiw7fw22cP8YaP/p+c4Ku2yDDuwS0ktz7O0Z9vZfvWfXRt7ab7wACvJjzW/PuHssFXXt08Ooc8Ogc9dvcMsWfXaPWrnp5hPtVRlw2+qmqpoaqlnqo5DcSbG3DqW4g0zsGpa8aP1zL8L5/Pecewhu9Eg0pXTiSKE4nxh709OcFXGQ1/yGr4XsrHS6az1a5qG6uywVeZJGuZoKqKiEM8Fgn2XYen+49mg68yGn64ylWwBKOWhspoTvCVO2bbEQLN3x0NzsqQT4MPt0Xc3OArR0b1+3DQVqF7jYdDrvZ+XFDVpO4Wum6cex537iTvfao1/DCq559eDFM2Tz9TX+QeEbnD7t+e8y7GPA5cAMGXBLAD+E3olH8wxjwwmYeWhNFXFEWZMozBn5rZO9cRpKqBoL7IE4wx+mO4EfiVMWboZB6q8o6iKEoIY4KRfjHLSZJTXwQoWF/EcgvwgzFtnxORF0TkPhGpKOahOtJXFEUZwySqYjWJyPrQ/lpbCwQAEXkUmJPnursm8z42Ff15wLpQ853AISBGUHvkduCfJrpXSRj9V/f38vX9LxN3hceuMoxseZijP9xG95b9vLK1m85DAxwaTtOV9BjwfBKhb+BNr30nu3oT7Nk2xM7ObezpGqSvd5jBY8Mk+pMMDw5lE6et/NAVxFubceubcetbsvq9H6/Fr5jFgC8MpgxDKR8nEsur3zvRGJFYkCzNicRwK+I8vuVIQf3eSwbFT8JFUFZcOJdYxKEq5hKLuFn9PqPphwufJAf7gOP1+7CuD0EBlPp4NEe/jzpOtthJvuInE2n6YWJOpsBJrn6f+SmZrwBKsbihi8ZefjLz6Qtdq5J5mWMmNYrvMsasLHwrc2WhYyIymfoiNwMPGmNSoXsftJsjIvId4OPFvLDKO4qiKGHsPP1ilpNkMvVFbmWMtGO/KJBgRHU9sKmYh5bESF9RFGWqMExZwrW89UVEZCXwPmPMbXa/A5gPPDnm+vtFpJngx+kG4H3FPFSNvqIoShhjSCdPv9E3xnSTp76IMWY9cFtofzcwL895l5/Ic9XoK4qihDAGfKNpGKaV5tkVfOK/v4GGFR3cu2oNPak0A55P0kbEuRI4EmsiDnMrozTEHJorIlQ1xLnt//6Rof4RRgYHAoftYB/e8CC+l8QbSWSrSwHM+qt7SFfOZiDlM5jySXg+iZRP31GPvpF+BkZsxasRj1lzF1kHbgw3FrcO3IpQVSs3mxxt784e0tZR6yXTGGOyla7GVrkyfppz563IqW6VXUJJ0qKOgyvgDQ8CuQ5byF/lqj4ezeuwHZsYrZggquMTrmXukeuwLVTpajII+Z2uJ1Ita+x9i0UTppUXaTX6iqIo5YEBzuB8a2r0FUVRxqIjfUVRlDLBN2Sl4zORkjD6/oKzefqvv8Cuo0PEeIBF1VEaYi6zGquoaopT3VpNdcssquY0UtVST6yxAbexDbe+mW23/TSr2YfJJkezAVRuJMYDu5L0jRwKtHubIK0vkSKR9Ogf9kiEAqzmLDsnq9lnCp44rt23BU4ywVRPrXshR7PPlyAtHEy1vG1WVrOPuME60PJHtzPJ0jJ+ibHka5tdEcnR7DMJ0sYWOMno15NJjBZxpWCRk5MtSOKOucGpLnAS3FM1e2UUlXcURVHKBINReUdRFKVcUEeuoihKmaFGf5rZvucwf/v3/xs/lWTgmbVQXR8kQaucTSoSZyjlk/AMvSmfV5Np+kY8+oZTDCTTxOtbj0uE5lYE60gsGujxdm79fQ+9ZJOh5SZA89M+ac8L9Hg7r/6qay7Mzp0fmwQtO8fedYg6wsPf35WTCC2sleebV7+koXrcRGjhYiXpZKKof0Pjp6mJBar72CRokH9e/WSIFaG7n+i8+nBBllPJZHR81ejLB2N09o6iKErZYNDZO4qiKGWDavqKoihlhso7iqIoZUKg6U/3W5w+SsLou7FKWs5ZjRtxuGKd4CWP4qU6baBUmrRn8D0/W43K+Ia05+F7Sd76zv9snasu8aibU31qbEKzT336u6EgKT9v9akMt150LY5wnKM1n+N1uK8re10xAU8LaoNSl8VUn5pMAFV1NLhTPp/kyQY8Rd3cG5xKv6d7mryo6pxVCqEjfUVRlDLBAFNSQmWaUKOvKIoSwmB09o6iKEq5EMzeUaM/rZx7VgO//9LbAai99P2Tuva7X7up6HM/2rmv6HNXz59V9Ln5Er6NR0v16flvqYqeaBmTiYmcjixoFtXelSnlDHfknj4rMA4icrWIbBORHSJyx3S8g6IoSj4yI/1ilpNBRG4Skc0i4tti6IXOy2svRWShiPxJRLaLyH+ISKyY50650RcRF/gK8JfAOcCtInLOVL+HoihKIdKmuOUk2QTcADxV6IQJ7OXngfuMMUuAHuC9xTx0Okb6q4Adxpidxpgk8EPguml4D0VRlOPwCdIwFLOcDMaYLcaYbROcltdeSjB/+3LgAXve94Dri3mumCl2WIjIjcDVxpjb7P67gUuMMR8cc94aYI3dPZfgW/FMoQnomvCs0uFM6w+ceX0qp/6cZYxpPtEbi8iv7f2LoRIYDu2vNcasneTzngA+boxZn+dYXnsJfAZ42hiz2LbPB35ljDl3oudNhyM3n1vuuG8e+w+3FkBE1htjCmpepYb2Z+ZzpvVJ+1M8xpirT9W9RORRYE6eQ3cZY35WzC3ytJlx2idkOoz+fmB+aL8dODAN76EoinJaMcZceZK3KGQvu4A6EYkYYzwmYUenQ9P/M7DEep5jwC3AQ9PwHoqiKDOdvPbSBLr848CN9rz3AMX8cph6o2+/lT4IrAO2AD8yxmye4LJJaWQlgPZn5nOm9Un7M8MQkf8iIvuBS4Ffisg62z5XRB6GCe3l7cBHRWQH0Ah8q6jnTrUjV1EURZk+piU4S1EURZke1OgriqKUETPa6JdqugYR+baIHBGRTaG2BhF5xIZMPyIi9bZdRORLto8viMiF0/fm+RGR+SLyuIhssWHjH7btJdknEakUkWdEZKPtz2dte96wdhGpsPs77PGO6Xz/QoiIKyLPi8gv7H6p92e3iLwoIhtEZL1tK8nP3Exixhr9Ek/X8F1g7FzfO4DHbMj0Y3Yfgv4tscsa4KtT9I6TwQM+ZoxZAbwe+ID9vyjVPo0AlxtjzgcuAK4WkddTOKz9vUCPDYS5z543E/kwgbMvQ6n3B+DNxpgLQnPyS/UzN3MwxszIhcCjvS60fydw53S/1yTevwPYFNrfBrTZ7TZgm93+OnBrvvNm6kIwNewtZ0KfgCrgOYIoxy4gYtuznz+CmROX2u2IPU+m+93H9KOdwAheDvyCIHinZPtj32030DSmreQ/c9O9zNiRPjAPCOc63m/bSpVWY8xBALtuse0l1U8rBbwO+BMl3CcrhWwAjgCPAK8AvSaYIge575ztjz3eRzBFbibxReATjBZ9aqS0+wNBhOlvRORZm5YFSvgzN1OYyfn0TzjMuMQomX6KSA3wE+AjxphjUjjR/YzvkzEmDVwgInXAg8CKfKfZ9Yzuj4i8HThijHlWRC7LNOc5tST6E2K1MeaAiLQAj4jI1nHOLZU+TTszeaR/pqVrOCwibQB2fcS2l0Q/RSRKYPDvN8b81DaXdJ8AjDG9wBMEvoo6EckMhMLvnO2PPV4LHJ3aNx2X1cC1IrKbIAvj5QQj/1LtDwDGmAN2fYTgi3kVZ8BnbrqZyUb/TEvX8BBBqDTkhkw/BPy1nX3weqAv8/N1piDBkP5bwBZjzL2hQyXZJxFptiN8RCQOXEngAC0U1h7u543Ab40VjmcCxpg7jTHtxpgOgr+T3xpj3kWJ9gdARKpFZFZmG3grQabdkvzMzSim26kw3gK8DXiZQG+9a7rfZxLv/QPgIJAiGIG8l0AzfQzYbtcN9lwhmKX0CvAisHK63z9Pf/4TwU/lF4ANdnlbqfYJeC3wvO3PJuBu23428AywA/gxUGHbK+3+Dnv87Onuwzh9uwz4Ran3x777Rrtszvz9l+pnbiYtmoZBURSljJjJ8o6iKIpyilGjryiKUkao0VcURSkj1OgriqKUEWr0FUVRygg1+sq0IyJpm0lxs818+VEROeHPpoh8MrTdIaFsp4pS7qjRV2YCCRNkUnwNQSK3twGfPon7fXLiUxSlPFGjr8woTBByvwb4oI2udEXkn0XkzzZP+t8BiMhlIvKUiDwoIi+JyNdExBGRe4C4/eVwv72tKyLfsL8kfmOjcBWlLFGjr8w4jDE7CT6bLQTRzH3GmIuBi4G/FZGF9tRVwMeA84BFwA3GmDsY/eXwLnveEuAr9pdEL/Bfp643ijKzUKOvzFQyWRPfSpBTZQNBOudGAiMO8IwxZqcJMmb+gCBdRD52GWM22O1nCWodKEpZMpNTKytlioicDaQJMigK8PfGmHVjzrmM41PnFsopMhLaTgMq7yhli470lRmFiDQDXwO+bILEUOuA/2FTOyMiS23WRYBVNgurA7wD+J1tT2XOVxQlFx3pKzOBuJVvogT1eP8VyKRw/iaBHPOcTfHcCVxvj/0RuIdA03+KIOc6wFrgBRF5DrhrKjqgKKWCZtlUShIr73zcGPP26X4XRSklVN5RFEUpI3SkryiKUkboSF9RFKWMUKOvKIpSRqjRVxRFKSPU6CuKopQRavQVRVHKiP8PlnhMidE46QAAAAFJREFUX9PsMY0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos_encoding = positional_encoding(50, 512)\n",
    "print (pos_encoding.shape)\n",
    "\n",
    "plt.pcolormesh(pos_encoding[0], cmap='RdBu')\n",
    "plt.xlabel('Depth')\n",
    "plt.xlim((0, 512))\n",
    "plt.ylabel('Position')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a_b4ou4TYqUN"
   },
   "source": [
    "## 遮挡（Masking）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s42Uydjkv0hF"
   },
   "source": [
    "遮挡一批序列中所有的填充标记（pad tokens）。这确保了模型不会将填充作为输入。该 mask 表明填充值 `0` 出现的位置：在这些位置 mask 输出 `1`，否则输出 `0`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U2i8-e1s8ti9"
   },
   "outputs": [],
   "source": [
    "def create_padding_mask(seq):\n",
    "    seq = tf.cast(tf.math.equal(seq, 0), tf.float32)\n",
    "\n",
    "    # 添加额外的维度来将填充加到\n",
    "    # 注意力对数（logits）。\n",
    "    return seq[:, tf.newaxis, tf.newaxis, :]  # (batch_size, 1, 1, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A7BYeBCNvi7n"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=207751, shape=(3, 1, 1, 5), dtype=float32, numpy=\n",
       "array([[[[0., 0., 1., 1., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0., 1., 1.]]],\n",
       "\n",
       "\n",
       "       [[[1., 1., 1., 0., 0.]]]], dtype=float32)>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.constant([[7, 6, 0, 0, 1], [1, 2, 3, 0, 0], [0, 0, 0, 4, 5]])\n",
    "create_padding_mask(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z0hzukDBgVom"
   },
   "source": [
    "前瞻遮挡（look-ahead mask）用于遮挡一个序列中的后续标记（future tokens）。换句话说，该 mask 表明了不应该使用的条目。\n",
    "\n",
    "这意味着要预测第三个词，将仅使用第一个和第二个词。与此类似，预测第四个词，仅使用第一个，第二个和第三个词，依此类推。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dVxS8OPI9uI0"
   },
   "outputs": [],
   "source": [
    "def create_look_ahead_mask(size):\n",
    "    #     tf.linalg.band_part(\n",
    "    #     input,\n",
    "    #     num_lower,\n",
    "    #     num_upper,\n",
    "    #     name=None\n",
    "    # )\n",
    "    # input:输入的张量.\n",
    "    # num_lower:下三角矩阵保留的副对角线数量，从主对角线开始计算，相当于下三角的带宽。取值为负数时，则全部保留。\n",
    "    # num_upper:上三角矩阵保留的副对角线数量，从主对角线开始计算，相当于上三角的带宽。取值为负数时，则全部保留。\n",
    "    mask = 1 - tf.linalg.band_part(tf.ones((size, size)), -1, 0)\n",
    "    return mask  # (seq_len, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yxKGuXxaBeeE"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=207766, shape=(3, 3), dtype=float32, numpy=\n",
       "array([[0., 1., 1.],\n",
       "       [0., 0., 1.],\n",
       "       [0., 0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.random.uniform((1, 3))\n",
    "temp = create_look_ahead_mask(x.shape[1])\n",
    "temp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xluDl5cXYy4y"
   },
   "source": [
    "## 按比缩放的点积注意力（Scaled dot product attention）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vsxEE_-Wa1gF"
   },
   "source": [
    "<img src=\"https://tensorflow.google.cn/images/tutorials/transformer/scaled_attention.png\" width=\"500\" alt=\"scaled_dot_product_attention\">\n",
    "\n",
    "Transformer 使用的注意力函数有三个输入：Q（请求（query））、K（主键（key））、V（数值（value））。用于计算注意力权重的等式为：\n",
    "\n",
    "$$\\Large{Attention(Q, K, V) = softmax_k(\\frac{QK^T}{\\sqrt{d_k}}) V} $$\n",
    "\n",
    "点积注意力被缩小了深度的平方根倍。这样做是因为对于较大的深度值，点积的大小会增大，从而推动 softmax 函数往仅有很小的梯度的方向靠拢，导致了一种很硬的（hard）softmax。\n",
    "\n",
    "例如，假设 `Q` 和 `K` 的均值为0，方差为1。它们的矩阵乘积将有均值为0，方差为 `dk`。因此，*`dk` 的平方根*被用于缩放（而非其他数值），因为，`Q` 和 `K` 的矩阵乘积的均值本应该为 0，方差本应该为1，这样会获得一个更平缓的 softmax。\n",
    "\n",
    "遮挡（mask）与 -1e9（接近于负无穷）相乘。这样做是因为遮挡与缩放的 Q 和 K 的矩阵乘积相加，并在 softmax 之前立即应用。目标是将这些单元归零，因为 softmax 的较大负数输入在输出中接近于零。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LazzUq3bJ5SH"
   },
   "outputs": [],
   "source": [
    "def scaled_dot_product_attention(q, k, v, mask):\n",
    "    \"\"\"计算注意力权重。\n",
    "  q, k, v 必须具有匹配的前置维度。\n",
    "  k, v 必须有匹配的倒数第二个维度，例如：seq_len_k = seq_len_v。\n",
    "  虽然 mask 根据其类型（填充或前瞻）有不同的形状，\n",
    "  但是 mask 必须能进行广播转换以便求和。\n",
    "  \n",
    "  参数:\n",
    "    q: 请求的形状 == (..., seq_len_q, depth)\n",
    "    k: 主键的形状 == (..., seq_len_k, depth)\n",
    "    v: 数值的形状 == (..., seq_len_v, depth_v)\n",
    "    mask: Float 张量，其形状能转换成\n",
    "          (..., seq_len_q, seq_len_k)。默认为None。\n",
    "    \n",
    "  返回值:\n",
    "    输出，注意力权重\n",
    "  \"\"\"\n",
    "\n",
    "    matmul_qk = tf.matmul(q, k,\n",
    "                          transpose_b=True)  # (..., seq_len_q, seq_len_k)\n",
    "\n",
    "    # 缩放 matmul_qk\n",
    "    dk = tf.cast(tf.shape(k)[-1], tf.float32)\n",
    "    scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)\n",
    "\n",
    "    # 将 mask 加入到缩放的张量上。\n",
    "    if mask is not None:\n",
    "        scaled_attention_logits += (mask * -1e9)\n",
    "\n",
    "    # softmax 在最后一个轴（seq_len_k）上归一化，因此分数\n",
    "    # 相加等于1。\n",
    "    attention_weights = tf.nn.softmax(scaled_attention_logits,\n",
    "                                      axis=-1)  # (..., seq_len_q, seq_len_k)\n",
    "\n",
    "    output = tf.matmul(attention_weights, v)  # (..., seq_len_q, depth_v)\n",
    "\n",
    "    return output, attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FiqETnhCkoXh"
   },
   "source": [
    "当 softmax 在 K 上进行归一化后，它的值决定了分配到 Q 的重要程度。\n",
    "\n",
    "输出表示注意力权重和 V（数值）向量的乘积。这确保了要关注的词保持原样，而无关的词将被清除掉。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n90YjClyInFy"
   },
   "outputs": [],
   "source": [
    "def print_out(q, k, v):\n",
    "    temp_out, temp_attn = scaled_dot_product_attention(q, k, v, None)\n",
    "    print('Attention weights are:')\n",
    "    print(temp_attn)\n",
    "    print('Output is:')\n",
    "    print(temp_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yAzUAf2DPlNt"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0. 1. 0. 0.]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[10.  0.]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "temp_k = tf.constant([[10,0,0],\n",
    "                      [0,10,0],\n",
    "                      [0,0,10],\n",
    "                      [0,0,10]], dtype=tf.float32)  # (4, 3)\n",
    "\n",
    "temp_v = tf.constant([[   1,0],\n",
    "                      [  10,0],\n",
    "                      [ 100,5],\n",
    "                      [1000,6]], dtype=tf.float32)  # (4, 2)\n",
    "\n",
    "# 这条 `请求（query）符合第二个`主键（key）`，\n",
    "# 因此返回了第二个`数值（value）`。\n",
    "temp_q = tf.constant([[0, 10, 0]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zg6k-fGhgXra"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0.  0.  0.5 0.5]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[550.    5.5]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# 这条请求符合重复出现的主键（第三第四个），\n",
    "# 因此，对所有的相关数值取了平均。\n",
    "temp_q = tf.constant([[0, 0, 10]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UAq3YOzUgXhb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0.5 0.5 0.  0. ]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[5.5 0. ]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# 这条请求符合第一和第二条主键，\n",
    "# 因此，对它们的数值去了平均。\n",
    "temp_q = tf.constant([[10, 10, 0]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aOz-4_XIhaTP"
   },
   "source": [
    "将所有请求一起*传递*。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6dlU8Tm-hYrF"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor(\n",
      "[[0.  0.  0.5 0.5]\n",
      " [0.  1.  0.  0. ]\n",
      " [0.5 0.5 0.  0. ]], shape=(3, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor(\n",
      "[[550.    5.5]\n",
      " [ 10.    0. ]\n",
      " [  5.5   0. ]], shape=(3, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "temp_q = tf.constant([[0, 0, 10], [0, 10, 0], [10, 10, 0]], dtype=tf.float32)  # (3, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kmzGPEy64qmA"
   },
   "source": [
    "## 多头注意力（Multi-head attention）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fz5BMC8Kaoqo"
   },
   "source": [
    "<img src=\"https://tensorflow.google.cn/images/tutorials/transformer/multi_head_attention.png\" width=\"500\" alt=\"multi-head attention\">\n",
    "\n",
    "\n",
    "多头注意力由四部分组成：\n",
    "*    线性层并分拆成多头。\n",
    "*    按比缩放的点积注意力。\n",
    "*    多头及联。\n",
    "*    最后一层线性层。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JPmbr6F1C-v_"
   },
   "source": [
    "每个多头注意力块有三个输入：Q（请求）、K（主键）、V（数值）。这些输入经过线性（Dense）层，并分拆成多头。 \n",
    "\n",
    "将上面定义的 `scaled_dot_product_attention` 函数应用于每个头（进行了广播（broadcasted）以提高效率）。注意力这步必须使用一个恰当的 mask。然后将每个头的注意力输出连接起来（用`tf.transpose` 和 `tf.reshape`），并放入最后的 `Dense` 层。\n",
    "\n",
    "Q、K、和 V 被拆分到了多个头，而非单个的注意力头，因为多头允许模型共同注意来自不同表示空间的不同位置的信息。在分拆后，每个头部的维度减少，因此总的计算成本与有着全部维度的单个注意力头相同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BSV3PPKsYecw"
   },
   "outputs": [],
   "source": [
    "class MultiHeadAttention(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, num_heads):\n",
    "        super(MultiHeadAttention, self).__init__()\n",
    "        self.num_heads = num_heads\n",
    "        self.d_model = d_model\n",
    "\n",
    "        assert d_model % self.num_heads == 0\n",
    "\n",
    "        self.depth = d_model // self.num_heads\n",
    "\n",
    "        self.wq = tf.keras.layers.Dense(d_model)\n",
    "        self.wk = tf.keras.layers.Dense(d_model)\n",
    "        self.wv = tf.keras.layers.Dense(d_model)\n",
    "\n",
    "        self.dense = tf.keras.layers.Dense(d_model)\n",
    "\n",
    "    def split_heads(self, x, batch_size):\n",
    "        \"\"\"分拆最后一个维度到 (num_heads, depth).\n",
    "    转置结果使得形状为 (batch_size, num_heads, seq_len, depth)\n",
    "    \"\"\"\n",
    "        x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))\n",
    "        return tf.transpose(x, perm=[0, 2, 1, 3])\n",
    "\n",
    "    def call(self, v, k, q, mask):\n",
    "        batch_size = tf.shape(q)[0]\n",
    "\n",
    "        q = self.wq(q)  # (batch_size, seq_len, d_model)\n",
    "        k = self.wk(k)  # (batch_size, seq_len, d_model)\n",
    "        v = self.wv(v)  # (batch_size, seq_len, d_model)\n",
    "\n",
    "        q = self.split_heads(\n",
    "            q, batch_size)  # (batch_size, num_heads, seq_len_q, depth)\n",
    "        k = self.split_heads(\n",
    "            k, batch_size)  # (batch_size, num_heads, seq_len_k, depth)\n",
    "        v = self.split_heads(\n",
    "            v, batch_size)  # (batch_size, num_heads, seq_len_v, depth)\n",
    "\n",
    "        # scaled_attention.shape == (batch_size, num_heads, seq_len_q, depth)\n",
    "        # attention_weights.shape == (batch_size, num_heads, seq_len_q, seq_len_k)\n",
    "        scaled_attention, attention_weights = scaled_dot_product_attention(\n",
    "            q, k, v, mask)\n",
    "\n",
    "        scaled_attention = tf.transpose(\n",
    "            scaled_attention,\n",
    "            perm=[0, 2, 1, 3])  # (batch_size, seq_len_q, num_heads, depth)\n",
    "\n",
    "        concat_attention = tf.reshape(\n",
    "            scaled_attention,\n",
    "            (batch_size, -1, self.d_model))  # (batch_size, seq_len_q, d_model)\n",
    "\n",
    "        output = self.dense(\n",
    "            concat_attention)  # (batch_size, seq_len_q, d_model)\n",
    "\n",
    "        return output, attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0D8FJue5lDyZ"
   },
   "source": [
    "创建一个 `MultiHeadAttention` 层进行尝试。在序列中的每个位置 `y`，`MultiHeadAttention` 在序列中的所有其他位置运行所有8个注意力头，在每个位置y，返回一个新的同样长度的向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Hu94p-_-2_BX"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([1, 60, 512]), TensorShape([1, 8, 60, 60]))"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp_mha = MultiHeadAttention(d_model=512, num_heads=8)\n",
    "y = tf.random.uniform((1, 60, 512))  # (batch_size, encoder_sequence, d_model)\n",
    "out, attn = temp_mha(y, k=y, q=y, mask=None)\n",
    "out.shape, attn.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RdDqGayx67vv"
   },
   "source": [
    "## 点式前馈网络（Point wise feed forward network）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gBqzJXGfHK3X"
   },
   "source": [
    "点式前馈网络由两层全联接层组成，两层之间有一个 ReLU 激活函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ET7xLt0yCT6Z"
   },
   "outputs": [],
   "source": [
    "def point_wise_feed_forward_network(d_model, dff):\n",
    "    return tf.keras.Sequential([\n",
    "        tf.keras.layers.Dense(dff,\n",
    "                              activation='relu'),  # (batch_size, seq_len, dff)\n",
    "        tf.keras.layers.Dense(d_model)  # (batch_size, seq_len, d_model)\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mytb1lPyOHLB"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 50, 512])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_ffn = point_wise_feed_forward_network(512, 2048)\n",
    "sample_ffn(tf.random.uniform((64, 50, 512))).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7e7hKcxn6-zd"
   },
   "source": [
    "## 编码与解码（Encoder and decoder）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yScbC0MUH8dS"
   },
   "source": [
    "<img src=\"https://tensorflow.google.cn/images/tutorials/transformer/transformer.png\" width=\"600\" alt=\"transformer\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MfYJG-Kvgwy2"
   },
   "source": [
    "Transformer 模型与标准的[具有注意力机制的序列到序列模型（sequence to sequence with attention model）](nmt_with_attention.ipynb)，遵循相同的一般模式。\n",
    "\n",
    "* 输入语句经过 `N` 个编码器层，为序列中的每个词/标记生成一个输出。\n",
    "* 解码器关注编码器的输出以及它自身的输入（自注意力）来预测下一个词。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QFv-FNYUmvpn"
   },
   "source": [
    "### 编码器层（Encoder layer）\n",
    "\n",
    "每个编码器层包括以下子层：\n",
    "\n",
    "1.   多头注意力（有填充遮挡）\n",
    "2.   点式前馈网络（Point wise feed forward networks）。\n",
    "\n",
    "每个子层在其周围有一个残差连接，然后进行层归一化。残差连接有助于避免深度网络中的梯度消失问题。\n",
    "\n",
    "每个子层的输出是 `LayerNorm(x + Sublayer(x))`。归一化是在 `d_model`（最后一个）维度完成的。Transformer 中有 N 个编码器层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ncyS-Ms3i2x_"
   },
   "outputs": [],
   "source": [
    "class EncoderLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
    "        super(EncoderLayer, self).__init__()\n",
    "\n",
    "        self.mha = MultiHeadAttention(d_model, num_heads)\n",
    "        self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    "\n",
    "        #     epsilon: Small float added to variance to avoid dividing by zero.\n",
    "        self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "\n",
    "        self.dropout1 = tf.keras.layers.Dropout(rate)\n",
    "        self.dropout2 = tf.keras.layers.Dropout(rate)\n",
    "\n",
    "    def call(self, x, training, mask):\n",
    "\n",
    "        attn_output, _ = self.mha(x, x, x,\n",
    "                                  mask)  # (batch_size, input_seq_len, d_model)\n",
    "        attn_output = self.dropout1(attn_output, training=training)\n",
    "        out1 = self.layernorm1(\n",
    "            x + attn_output)  # (batch_size, input_seq_len, d_model)\n",
    "\n",
    "        ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, d_model)\n",
    "        ffn_output = self.dropout2(ffn_output, training=training)\n",
    "        out2 = self.layernorm2(\n",
    "            out1 + ffn_output)  # (batch_size, input_seq_len, d_model)\n",
    "\n",
    "        return out2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AzZRXdO0mI48"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 43, 512])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_encoder_layer = EncoderLayer(512, 8, 2048)\n",
    "\n",
    "sample_encoder_layer_output = sample_encoder_layer(\n",
    "    tf.random.uniform((64, 43, 512)), False, None)\n",
    "\n",
    "sample_encoder_layer_output.shape  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6LO_48Owmx_o"
   },
   "source": [
    "### 解码器层（Decoder layer）\n",
    "\n",
    "每个解码器层包括以下子层：\n",
    "\n",
    "1.   遮挡的多头注意力（前瞻遮挡和填充遮挡）\n",
    "2.   多头注意力（用填充遮挡）。V（数值）和 K（主键）接收*编码器输出*作为输入。Q（请求）接收*遮挡的多头注意力子层的输出*。\n",
    "3.   点式前馈网络\n",
    "\n",
    "每个子层在其周围有一个残差连接，然后进行层归一化。每个子层的输出是 `LayerNorm(x + Sublayer(x))`。归一化是在 `d_model`（最后一个）维度完成的。\n",
    "\n",
    "Transformer 中共有 N 个解码器层。\n",
    "\n",
    "当 Q 接收到解码器的第一个注意力块的输出，并且 K 接收到编码器的输出时，注意力权重表示根据编码器的输出赋予解码器输入的重要性。换一种说法，解码器通过查看编码器输出和对其自身输出的自注意力，预测下一个词。参看按比缩放的点积注意力部分的演示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9SoX0-vd1hue"
   },
   "outputs": [],
   "source": [
    "class DecoderLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
    "        super(DecoderLayer, self).__init__()\n",
    "\n",
    "        self.mha1 = MultiHeadAttention(d_model, num_heads)\n",
    "        self.mha2 = MultiHeadAttention(d_model, num_heads)\n",
    "\n",
    "        self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    "\n",
    "        self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm3 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "\n",
    "        self.dropout1 = tf.keras.layers.Dropout(rate)\n",
    "        self.dropout2 = tf.keras.layers.Dropout(rate)\n",
    "        self.dropout3 = tf.keras.layers.Dropout(rate)\n",
    "\n",
    "    def call(self, x, enc_output, training, look_ahead_mask, padding_mask):\n",
    "        # enc_output.shape == (batch_size, input_seq_len, d_model)\n",
    "\n",
    "        attn1, attn_weights_block1 = self.mha1(\n",
    "            x, x, x, look_ahead_mask)  # (batch_size, target_seq_len, d_model)\n",
    "        attn1 = self.dropout1(attn1, training=training)\n",
    "        out1 = self.layernorm1(attn1 + x)\n",
    "\n",
    "        attn2, attn_weights_block2 = self.mha2(\n",
    "            enc_output, enc_output, out1,\n",
    "            padding_mask)  # (batch_size, target_seq_len, d_model)\n",
    "        attn2 = self.dropout2(attn2, training=training)\n",
    "        out2 = self.layernorm2(attn2 +\n",
    "                               out1)  # (batch_size, target_seq_len, d_model)\n",
    "\n",
    "        ffn_output = self.ffn(out2)  # (batch_size, target_seq_len, d_model)\n",
    "        ffn_output = self.dropout3(ffn_output, training=training)\n",
    "        out3 = self.layernorm3(ffn_output +\n",
    "                               out2)  # (batch_size, target_seq_len, d_model)\n",
    "\n",
    "        return out3, attn_weights_block1, attn_weights_block2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ne2Bqx8k71l0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 50, 512])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_decoder_layer = DecoderLayer(512, 8, 2048)\n",
    "\n",
    "sample_decoder_layer_output, _, _ = sample_decoder_layer(\n",
    "    tf.random.uniform((64, 50, 512)), sample_encoder_layer_output, False, None,\n",
    "    None)\n",
    "\n",
    "sample_decoder_layer_output.shape  # (batch_size, target_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SE1H51Ajm0q1"
   },
   "source": [
    "### 编码器（Encoder）\n",
    "\n",
    "`编码器` 包括：\n",
    "1.   输入嵌入（Input Embedding）\n",
    "2.   位置编码（Positional Encoding）\n",
    "3.   N 个编码器层（encoder layers）\n",
    "\n",
    "输入经过嵌入（embedding）后，该嵌入与位置编码相加。该加法结果的输出是编码器层的输入。编码器的输出是解码器的输入。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jpEox7gJ8FCI"
   },
   "outputs": [],
   "source": [
    "class Encoder(tf.keras.layers.Layer):\n",
    "    def __init__(self,\n",
    "                 num_layers,\n",
    "                 d_model,\n",
    "                 num_heads,\n",
    "                 dff,\n",
    "                 input_vocab_size,\n",
    "                 maximum_position_encoding,\n",
    "                 rate=0.1):\n",
    "        super(Encoder, self).__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.num_layers = num_layers\n",
    "\n",
    "        self.embedding = tf.keras.layers.Embedding(input_vocab_size, d_model)\n",
    "        self.pos_encoding = positional_encoding(maximum_position_encoding,\n",
    "                                                self.d_model)\n",
    "\n",
    "        self.enc_layers = [\n",
    "            EncoderLayer(d_model, num_heads, dff, rate)\n",
    "            for _ in range(num_layers)\n",
    "        ]\n",
    "\n",
    "        self.dropout = tf.keras.layers.Dropout(rate)\n",
    "\n",
    "    def call(self, x, training, mask):\n",
    "\n",
    "        seq_len = tf.shape(x)[1]\n",
    "\n",
    "        # 将嵌入和位置编码相加。\n",
    "        x = self.embedding(x)  \n",
    "        # (batch_size, input_seq_len, d_model)\n",
    "        #     为什么要乘以embedding size的开方？\n",
    "        # 猜测是因为embedding matrix的初始化方式是xavier init，这种方式的方差是1/embedding size，\n",
    "        # 因此乘以embedding size的开方使得embedding matrix的方差是1，在这个scale下可能更有利于embedding matrix的收敛。\n",
    "        x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "        x += self.pos_encoding[:, :seq_len, :]\n",
    "\n",
    "        x = self.dropout(x, training=training)\n",
    "\n",
    "        for i in range(self.num_layers):\n",
    "            x = self.enc_layers[i](x, training, mask)\n",
    "\n",
    "        return x  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8QG9nueFQKXx"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 62, 512)\n"
     ]
    }
   ],
   "source": [
    "sample_encoder = Encoder(num_layers=2, d_model=512, num_heads=8, \n",
    "                         dff=2048, input_vocab_size=8500,\n",
    "                         maximum_position_encoding=10000)\n",
    "\n",
    "sample_encoder_output = sample_encoder(tf.random.uniform((64, 62)), \n",
    "                                       training=False, mask=None)\n",
    "\n",
    "print (sample_encoder_output.shape)  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p-uO6ls8m2O5"
   },
   "source": [
    "### 解码器（Decoder）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZtT7PKzrXkNr"
   },
   "source": [
    "`解码器`包括：\n",
    "1.   输出嵌入（Output Embedding）\n",
    "2.   位置编码（Positional Encoding）\n",
    "3.   N 个解码器层（decoder layers）\n",
    "\n",
    "目标（target）经过一个嵌入后，该嵌入和位置编码相加。该加法结果是解码器层的输入。解码器的输出是最后的线性层的输入。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d5_d5-PLQXwY"
   },
   "outputs": [],
   "source": [
    "class Decoder(tf.keras.layers.Layer):\n",
    "    def __init__(self,\n",
    "                 num_layers,\n",
    "                 d_model,\n",
    "                 num_heads,\n",
    "                 dff,\n",
    "                 target_vocab_size,\n",
    "                 maximum_position_encoding,\n",
    "                 rate=0.1):\n",
    "        super(Decoder, self).__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.num_layers = num_layers\n",
    "\n",
    "        self.embedding = tf.keras.layers.Embedding(target_vocab_size, d_model)\n",
    "        self.pos_encoding = positional_encoding(maximum_position_encoding,\n",
    "                                                d_model)\n",
    "\n",
    "        self.dec_layers = [\n",
    "            DecoderLayer(d_model, num_heads, dff, rate)\n",
    "            for _ in range(num_layers)\n",
    "        ]\n",
    "        self.dropout = tf.keras.layers.Dropout(rate)\n",
    "\n",
    "    def call(self, x, enc_output, training, look_ahead_mask, padding_mask):\n",
    "\n",
    "        seq_len = tf.shape(x)[1]\n",
    "        attention_weights = {}\n",
    "\n",
    "        x = self.embedding(x)  # (batch_size, target_seq_len, d_model)\n",
    "        x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "        x += self.pos_encoding[:, :seq_len, :]\n",
    "\n",
    "        x = self.dropout(x, training=training)\n",
    "\n",
    "        for i in range(self.num_layers):\n",
    "            x, block1, block2 = self.dec_layers[i](x, enc_output, training,\n",
    "                                                   look_ahead_mask,\n",
    "                                                   padding_mask)\n",
    "\n",
    "            attention_weights['decoder_layer{}_block1'.format(i + 1)] = block1\n",
    "            attention_weights['decoder_layer{}_block2'.format(i + 1)] = block2\n",
    "\n",
    "        # x.shape == (batch_size, target_seq_len, d_model)\n",
    "        return x, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a1jXoAMRZyvu"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([64, 26, 512]), TensorShape([64, 8, 26, 62]))"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_decoder = Decoder(num_layers=2, d_model=512, num_heads=8, \n",
    "                         dff=2048, target_vocab_size=8000,\n",
    "                         maximum_position_encoding=5000)\n",
    "\n",
    "output, attn = sample_decoder(tf.random.uniform((64, 26)), \n",
    "                              enc_output=sample_encoder_output, \n",
    "                              training=False, look_ahead_mask=None, \n",
    "                              padding_mask=None)\n",
    "\n",
    "output.shape, attn['decoder_layer2_block2'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y54xnJnuYgJ7"
   },
   "source": [
    "## 创建 Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uERO1y54cOKq"
   },
   "source": [
    "Transformer 包括编码器，解码器和最后的线性层。解码器的输出是线性层的输入，返回线性层的输出。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PED3bIpOYkBu"
   },
   "outputs": [],
   "source": [
    "class Transformer(tf.keras.Model):\n",
    "    def __init__(self,\n",
    "                 num_layers,\n",
    "                 d_model,\n",
    "                 num_heads,\n",
    "                 dff,\n",
    "                 input_vocab_size,\n",
    "                 target_vocab_size,\n",
    "                 pe_input,\n",
    "                 pe_target,\n",
    "                 rate=0.1):\n",
    "        super(Transformer, self).__init__()\n",
    "\n",
    "        self.encoder = Encoder(num_layers, d_model, num_heads, dff,\n",
    "                               input_vocab_size, pe_input, rate)\n",
    "\n",
    "        self.decoder = Decoder(num_layers, d_model, num_heads, dff,\n",
    "                               target_vocab_size, pe_target, rate)\n",
    "\n",
    "        self.final_layer = tf.keras.layers.Dense(target_vocab_size)\n",
    "\n",
    "    def call(self, inp, tar, training, enc_padding_mask, look_ahead_mask,\n",
    "             dec_padding_mask):\n",
    "\n",
    "        enc_output = self.encoder(\n",
    "            inp, training,\n",
    "            enc_padding_mask)  # (batch_size, inp_seq_len, d_model)\n",
    "\n",
    "        # dec_output.shape == (batch_size, tar_seq_len, d_model)\n",
    "        dec_output, attention_weights = self.decoder(tar, enc_output, training,\n",
    "                                                     look_ahead_mask,\n",
    "                                                     dec_padding_mask)\n",
    "\n",
    "        final_output = self.final_layer(\n",
    "            dec_output)  # (batch_size, tar_seq_len, target_vocab_size)\n",
    "\n",
    "        return final_output, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tJ4fbQcIkHW1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 26, 8000])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_transformer = Transformer(\n",
    "    num_layers=2, d_model=512, num_heads=8, dff=2048, \n",
    "    input_vocab_size=8500, target_vocab_size=8000, \n",
    "    pe_input=10000, pe_target=6000)\n",
    "\n",
    "temp_input = tf.random.uniform((64, 62))\n",
    "temp_target = tf.random.uniform((64, 26))\n",
    "\n",
    "fn_out, _ = sample_transformer(temp_input, temp_target, training=False, \n",
    "                               enc_padding_mask=None, \n",
    "                               look_ahead_mask=None,\n",
    "                               dec_padding_mask=None)\n",
    "\n",
    "fn_out.shape  # (batch_size, tar_seq_len, target_vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wsINyf1VEQLC"
   },
   "source": [
    "## 配置超参数（hyperparameters）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zVjWCxFNcgbt"
   },
   "source": [
    "为了让本示例小且相对较快，已经减小了*num_layers、 d_model 和  dff* 的值。 \n",
    "\n",
    "Transformer 的基础模型使用的数值为：*num_layers=6*，*d_model = 512*，*dff = 2048*。关于所有其他版本的 Transformer，请查阅[论文](https://arxiv.org/abs/1706.03762)。\n",
    "\n",
    "Note：通过改变以下数值，您可以获得在许多任务上达到最先进水平的模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lnJn5SLA2ahP"
   },
   "outputs": [],
   "source": [
    "num_layers = 4\n",
    "d_model = 128\n",
    "dff = 512\n",
    "num_heads = 8\n",
    "\n",
    "input_vocab_size = tokenizer_pt.vocab_size + 2\n",
    "target_vocab_size = tokenizer_en.vocab_size + 2\n",
    "dropout_rate = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xYEGhEOtzn5W"
   },
   "source": [
    "## 优化器（Optimizer）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GOmWW--yP3zx"
   },
   "source": [
    "根据[论文](https://arxiv.org/abs/1706.03762)中的公式，将 Adam 优化器与自定义的学习速率调度程序（scheduler）配合使用。\n",
    "\n",
    "$$\\Large{lrate = d_{model}^{-0.5} * min(step{\\_}num^{-0.5}, step{\\_}num * warmup{\\_}steps^{-1.5})}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iYQdOO1axwEI"
   },
   "outputs": [],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super(CustomSchedule, self).__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "\n",
    "        self.warmup_steps = warmup_steps\n",
    "\n",
    "    def __call__(self, step):\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps**-1.5)\n",
    "\n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7r4scdulztRx"
   },
   "outputs": [],
   "source": [
    "learning_rate = CustomSchedule(d_model)\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98, \n",
    "                                     epsilon=1e-9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f33ZCgvHpPdG"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,0,'Train Step')"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VfWZ+PHPk4QkJCEJ2SCsYZegiBrR4r6j00o7asXaDtNqmbbazV9bdfqbtuOMv9Euaq3aDqO0aLVItbbYca/WrbJEVGQpkHtBCFtuWAIJEEjy/P4438Al3pvcJPfk3iTP+/XKK+ee+z3f89wTyJNzvt/zHFFVjDHGmHhLSXQAxhhj+iZLMMYYY3xhCcYYY4wvLMEYY4zxhSUYY4wxvrAEY4wxxheWYIwxxvjCEowxxhhfWIIxxhjji7REB5BIRUVFWlZWlugwjDGmV3n33XdrVbW4o3b9OsGUlZVRWVmZ6DCMMaZXEZGPYmlnl8iMMcb4whKMMcYYX1iCMcYY4wtLMMYYY3zha4IRkZkisk5EqkTktgjvZ4jIk+79pSJSFvbe7W79OhG5LGz9fBGpEZFVUfb5HRFRESny4zMZY4yJjW8JRkRSgQeBy4Fy4DoRKW/T7AZgj6qOB+4F7nbblgOzgSnATOAh1x/Ab9y6SPscCVwCbI7rhzHGGNNpfp7BTAeqVDWoqoeBhcCsNm1mAQvc8lPARSIibv1CVW1U1Y1AlesPVX0D2B1ln/cC3wPsMZ3GGJNgfiaY4cCWsNfVbl3ENqraBNQBhTFuexwRuRLYqqofdC/s5KWqLFq+hfrGpkSHYowxHfIzwUiEdW3PLKK1iWXbY52IZAHfB37QYVAic0WkUkQqQ6FQR82Tyvtb9vK9p1dy61MrEx2KMcZ0yM8EUw2MDHs9AtgWrY2IpAF5eJe/Ytk23DhgDPCBiGxy7VeIyNC2DVV1nqpWqGpFcXGHlQ6SyubdBwB4ec3OBEdijDEd8zPBLAcmiMgYEUnHG7Rf3KbNYmCOW74aeFVV1a2f7WaZjQEmAMui7UhVP1TVElUtU9UyvAR1qqruiO9HSqxAqAGAw80tbN51IMHRGGNM+3xLMG5M5WbgRWAtsEhVV4vIHW68BOARoFBEqoBbgNvctquBRcAa4AXgJlVtBhCR3wHvAJNEpFpEbvDrMySbQKgecRcPX1i9PbHBGGNMB8Q7YeifKioqtDcVu7z8528yJDeD0P5G0tNSeOZrZyU6JGNMPyQi76pqRUft7E7+XqKlRdlYW8+44hyuOKmU9zbvZXvdwUSHZYwxUVmC6SW21R3k0JEWxhZnM/NEb+7CC6v61BCTMaaPsQTTSwTdAP+44hzGFecwacggnv2gvYl1xhiTWJZgeolAqB6AscXZAMw6ZRgrNu/lo10NiQzLGGOisgTTSwRDDQzKTKM4JwOAT0/zChv88T07izHGJCdLML1EIFTP2OIcxM1THpY/kDPHFvDMe9X055mAxpjkZQmmlwiGGhhXlH3cun88ZQSbdh3gvS17ExSVMcZEZwmmF6hvbGLHvkOMK8k5bv3lJw0lIy2FZ1ZsTVBkxhgTnSWYXmCjm0E2ts0ZzKDMAVxSPoRnV26jsak5EaEZY0xUlmB6gWCtN4Os7RkMwDUVI9l74AgvrbYCmMaY5GIJphcI1NSTIjC6MOtj750zvogRgwfyxFJ7iKcxJrlYgukFArUNjBicRUZa6sfeS0kRrps+ineCuwi6e2WMMSYZWILpBQI19Ywrzo76/jUVI0hLERYu3xK1jTHG9DRLMEmupUXZtKuBscUfH39pVTIok4snD+Gpd6ttsN8YkzQswSS51iKX49pJMACfO2MUuxsOWwFMY0zSsAST5FqfYjm2nUtkAGePL2JMUTbz395kd/YbY5KCJZgk1zpw39EZTEqK8MWzyvhgy15WbN7TE6EZY0y7LMEkuUConkGZaRTlpHfY9qpTR5CbmcYjb23sgciMMaZ9lmCSXDDUcFyRy/ZkZ6Rx3RmjeGHVDrbsPtAD0RljTHSWYJJcMNTQ7hTltv55RhkpIvzmb5v8C8oYY2Lga4IRkZkisk5EqkTktgjvZ4jIk+79pSJSFvbe7W79OhG5LGz9fBGpEZFVbfr6iYj8XURWisgzIpLv52frCUeLXHYw/hKuNG8gV5xUypPLt7D3wGEfozPGmPb5lmBEJBV4ELgcKAeuE5HyNs1uAPao6njgXuBut205MBuYAswEHnL9AfzGrWvrZeBEVZ0KrAduj+sHSoCNRx+THPsZDMBXzx9HfWOTncUYYxLKzzOY6UCVqgZV9TCwEJjVps0sYIFbfgq4SLzBhlnAQlVtVNWNQJXrD1V9A9jddmeq+pKqNrmXS4AR8f5APe3YY5JjP4MBmFyay8WThzD/rY3sP3TEj9CMMaZDfiaY4UB47ZJqty5iG5cc6oDCGLdtz5eA5yO9ISJzRaRSRCpDoVAnuux5wVD0Ipcd+cZF49l3qInHlnzkQ2TGGNMxPxNMpGlPbe8AjNYmlm0j71Tk+0AT8Hik91V1nqpWqGpFcXFxLF0mTCDUwMiCyEUuOzJ1RD7nTSzm4Tc3cuBwU8cbGGNMnPmZYKqBkWGvRwDborURkTQgD+/yVyzbfoyIzAE+CVyvfeB29kCo/mMPGeuMr184nt0Nh/mtncUYYxLAzwSzHJggImNEJB1v0H5xmzaLgTlu+WrgVZcYFgOz3SyzMcAEYFl7OxORmcCtwJWq2utvAmlpUTbWNnRqBllbFWUFnDOhiF/+NcA+G4sxxvQw3xKMG1O5GXgRWAssUtXVInKHiFzpmj0CFIpIFXALcJvbdjWwCFgDvADcpKrNACLyO+AdYJKIVIvIDa6vB4BBwMsi8r6I/Mqvz9YTtu49SGNTS6cH+Nu6deYJ7DlwhP95IxinyIwxJjZpfnauqs8Bz7VZ94Ow5UPANVG2vRO4M8L666K0H9+tYJNMsLZrU5TbOnF4Hp+cWsrDb27kC58YTcmgzHiEZ4wxHbI7+ZNUoKZrU5Qj+c6lkzjS3MIv/lLV7b6MMSZWlmCSVLA29iKXHSkryuba00fyu2Wb2ejOjIwxxm+WYJKUV4MstiKXsfjmxRPISEvhzv9dE5f+jDGmI5ZgklQgVN/hQ8Y6o2RQJt+4aAKvrK3htXU1cevXGGOisQSThOobm9i5r7FbU5Qj+eJZYxhblM1/PLuGw00tce3bGGPasgSThI49xTJ+ZzAA6Wkp/NunygnWNvCbv9lDyYwx/rIEk4SCropyPGaQtXXBpBIuPKGEn7+yge11B+PevzHGtLIEk4QC3ShyGYsffqqcZlX+7Y+r6QMVdYwxScoSTBIKdqPIZSxGF2bz7Ysn8sranTy/aocv+zDGGEswSSgQqo/7AH9bN5w9hinDcvnBn1ZTd8DqlBlj4s8STJJpLXLZnSrKsUhLTeHuq6ay58Bh/t9za33dlzGmf7IEk2Rai1yOK/H3DAa8OmU3njOGJyu38Nrf7d4YY0x8WYJJMkcfk+zzGUyrb188kROGDuK7T61kV31jj+zTGNM/WIJJMn5OUY4kc0Aq9147jX0Hj3D7Hz60WWXGmLixBJNkgrX15MapyGWsJpfm8t3LJvHSmp38vrK6x/ZrjOnbLMEkmUBNA2PjWOQyVjecPYZPjC3k359dffQynTHGdIclmCQTrPV/inIkKSnCPdeeTMaAVL722xUcPNzc4zEYY/oWSzBJZP+hI+zc1xjXKsqdUZo3kPuuncb6mv383z+usvEYY0y3WIJJIhvj9Jjk7jh3YjFfv3ACT6+oZlHlloTFYYzp/XxNMCIyU0TWiUiViNwW4f0MEXnSvb9URMrC3rvdrV8nIpeFrZ8vIjUisqpNXwUi8rKIbHDfB/v52fwQOFpFuecvkYX75kUTOHt8Ef/2p9Ws2lqX0FiMMb2XbwlGRFKBB4HLgXLgOhEpb9PsBmCPqo4H7gXudtuWA7OBKcBM4CHXH8Bv3Lq2bgP+oqoTgL+4171KMNRAisAon4pcxio1Rbhv9jSKstP58qOV1Ow7lNB4jDG9k59nMNOBKlUNquphYCEwq02bWcACt/wUcJF406dmAQtVtVFVNwJVrj9U9Q1gd4T9hfe1APh0PD9MTwiGGhjlY5HLzijKyeB/5lSw98AR5j72LoeO2KC/MaZz/Ewww4Hwi/jVbl3ENqraBNQBhTFu29YQVd3u+toOlERqJCJzRaRSRCpDoVCMH6VneI9JTuzlsXBThuVx77Un8/6Wvdz29Eob9DfGdIqfCSbSjRxtf0NFaxPLtl2iqvNUtUJVK4qLi+PRZVw0uyKXiRzgj2TmiaV859KJ/PH9bfzi1apEh2OM6UX8TDDVwMiw1yOAbdHaiEgakId3+SuWbdvaKSKlrq9SoFdVb9zmilwm0xlMq5suGM8/njqce15ez8JlmxMdjjGml/AzwSwHJojIGBFJxxu0X9ymzWJgjlu+GnhVveswi4HZbpbZGGACsKyD/YX3NQf4Uxw+Q4/p6SKXnSEi3H3VVM6bWMy/PvMhL6/ZmeiQjDG9gG8Jxo2p3Ay8CKwFFqnqahG5Q0SudM0eAQpFpAq4BTfzS1VXA4uANcALwE2q2gwgIr8D3gEmiUi1iNzg+roLuERENgCXuNe9RmuRy54o098VA1JTeOj6UzlpeB43P7GC5ZsizbMwxphjpD8P3FZUVGhlZWWiwwDg+898yLMfbOODH17a43XIOmNXfSPX/OodausbeeLLZ3Li8LxEh2SM6WEi8q6qVnTUzu7kTxLBUAPjSnq+yGVnFeZksOBL08nJSOPzjyxl7fZ9iQ7JGJOkLMEkiUConrFFyXl5rK2RBVn8bu6ZZKalcv3DS1m3Y3+iQzLGJCFLMElg/6Ej1OxPXJHLrhhdmM3v5p5JWopw/cNLqKqxJGOMOV5MCUZEzhaRL7rlYjezy8TJ0QH+JJyi3J4xRV6SAWH2vCWs2WaXy4wxx3SYYETkh8CtwO1u1QDgt34G1d8Ea1uLXPaeM5hW44pzePJfziQ9NYVr571Dpc0uM8Y4sZzBfAa4EmgAUNVtwCA/g+pvgqEGUlMk4UUuu2pccQ6//+oMinIy+PwjS3l9fXKV4DHGJEYsCeawu/lRAUSk9/2ZneQCoXpGDh6YFEUuu2p4/kAW/csnGFOUw40LlvPnlR0VXjDG9HWxJJhFIvLfQL6IfBl4BXjY37D6l2CoodeNv0RSPCiDhXPP5OQR+dz8xHv86vWAFcg0ph/rMMGo6k/xSuk/DUwCfqCq9/sdWH/R3KIEaxt61Qyy9uQNHMBvbzyDT04t5a7n/86/PvMhR5pbEh2WMSYB0jpqICJ3q+qtwMsR1plu2rb3IIeTtMhlV2UOSOX+2acwujCLB18LUL3nIA9efyq5mQMSHZoxpgfFconskgjrLo93IP1VsjwmOd5SUoTvXnYCP75qKu8EdnHVQ38j6D6rMaZ/iJpgROSrIvIhXlHJlWFfG4GVPRdi3xZw98D0lUtkbX329JE8+qXp1NY3MuuBt60SszH9SHtnME8An8Irg/+psK/TVPXzPRBbvxAM1ZM3cACF2emJDsU3M8YX8ezXz6asKJsvP1rJPS+vp6XFBv+N6euiJhhVrVPVTap6nap+BBzEm6qcIyKjeizCPs57THJ20he57K4Rg7P4/Vc+wdWnjeD+v2zghgXL2dNwONFhGWN8FMud/J9yz1jZCLwObAKe9zmufiMYaug1RS67K3NAKj+5eir/8ekTebtqF5f//E2WBHclOixjjE9iGeT/T+BMYL2qjgEuAt72Nap+orXI5biSvjn+EomI8IUzR/OHr81gYHoq1/3PEu55aR1NNpXZmD4nlgRzRFV3ASkikqKqrwHTfI6rX2gtctlfzmDCnTg8jz9//WyuOnUE979axbXzllC950CiwzLGxFEsCWaviOQAbwCPi8jPgSZ/w+ofWotcju9HZzDhsjPS+Ok1J/Pz2dNYt2M/M+97kyeXb7a7/43pI2JJMLOAA8C3gReAAN5sMtNNgRpX5LKgfyaYVrOmDef5b57DicNzufXpD/nnXy9ne93BRIdljOmmWErFNKhqi6o2qeoC4EFgZiydi8hMEVknIlUicluE9zNE5En3/lIRKQt773a3fp2IXNZRnyJykYisEJH3ReQtERkfS4yJFKytZ1RBFulp9ty3kQVZPHHjmfz7lVNYtnE3l977Bosqt9jZjDG9WHs3Wua6X/IPiMil4rkZCAKf7ahjEUnFS0aXA+XAdSJS3qbZDcAeVR0P3Avc7bYtB2YDU/CS2UMiktpBn78ErlfVaXj38Pzf2A5B4gRqGhhb1L/PXsKlpAhzZpTxwrfOYfLQXL731Er+af4yNtU2JDo0Y0wXtPen82N4xS0/BG4EXgKuAWap6qwY+p4OVKlqUFUPAwvxLreFmwUscMtPAReJd0PILGChqjaq6kagyvXXXp8K5LrlPCCp68U3tygbd/WdIpfxNLowm4Vzz+SOWVN4f/NeLr3vDX7+ygYam5oTHZoxphPaK3Y5VlVPAhCRh4FaYJSqxvrw9eHAlrDX1cAZ0dqoapOI1AGFbv2SNtsOd8vR+rwReE5EDgL78KZWJ63WIpd9rQZZvKSkCP/0iTIumzKU//jzGu59ZT1/en8r//npE5kxvijR4RljYtDeGcyR1gVVbQY2diK5AES6Nb3tBfVobTq7HrxJCFeo6gjg18A9EYMSmSsilSJSGQol7smLVa7wY1+qouyHIbmZPPC5U1nwpek0q/K5h5dy8xMrbEqzMb1AewnmZBHZ5772A1Nbl0VkXwx9VwMjw16P4OOXrY62EZE0vEtbu9vZNuJ6ESkGTlbVpW79k8CMSEGp6jxVrVDViuLi4hg+hj9a74EZZ5fIYnLexGJe/Na5fPOiCbyydicX/ux1fvLi36lvtBnzxiSr9mqRpapqrvsapKppYcu50bYLsxyYICJjRCQdb9B+cZs2i4E5bvlq4FX3eObFwGw3y2wMMAFY1k6fe4A8EZno+roEWBvLAUiUgCtyWdCHi1zGW+aAVL59yURe/T/nc8WJQ3nwtQAX/PSvLFq+hWYrnmlM0vFtfqyqNgE3Ay/i/bJfpKqrReQOEbnSNXsEKBSRKuAW4Da37WpgEbAG796bm1S1OVqfbv2XgadF5APgC8B3/fps8RDsJ0Uu/TAsfyD3zT6FZ742g5GDB/K9p1fyyV+8xat/32nTmo1JItKf/0NWVFRoZWVlQvZ9+p2vcN7EYn56zckJ2X9foao8u3I7P3tpHR/tOsBpowfz3csmcebYwkSHZkyfJSLvqmpFR+3sDr8E2H/oCKH9jTZFOQ5EhCtPHsYrt5zHnZ85keo9B5g9bwlfeGQpK6v3Jjo8Y/o1SzAJcGyA32aQxcuA1BSuP2M0r3/3Av71ihP4cGsdVz7wNjcuWM57m/ckOjxj+qVYngezP2w2WevXFhF5RkTG9kSQfU3ATVG2GWTxlzkglbnnjuPN713Aty+eyPJNe/jMQ3/j8w8vZUlwl43RGNOD2rvRstU9eFOEn8C7D2U2MBRYB8wHzvcruL4qGLIil34blDmAb148gRvOGcNvl3zEw28GmT1vCRWjB3PTheM5f2KxTbAwxmexXCKbqar/rar7VXWfqs7Du6HxSWCwz/H1SYGQFbnsKTkZaXzlvHG8deuF/PuVU9i69yBf/PVyrrj/LZ56t9rKzxjjo1h+w7WIyGdFJMV9hRe6tOsNXeA9JtnOXnpS5oBU5swo4/XvXsCPr5pKc0sL3/n9B5x112vc/5cN7KpvTHSIxvQ5sSSY6/HuK6kBdrrlz4vIQLx7UkwntBa5HFdiA/yJkJ6WwmdPH8mL3zqXR780nSnDcrnn5fXMuOtVbnt6Jet3dqYakjGmPR2OwahqkOgPGHsrvuH0fVv3eEUu7QwmsUSEcycWc+7EYqpq9vPIW5v4w4pqFi7fwifGFnL9maO4tHyoXcY0phs6TDCuzteXgbLw9qr6Jf/C6rsC7jHJdgaTPMaXDOK//vEkvnvZJH63bDNPLN3MzU+8R1FOOp+tGMl100cxsiAr0WEa0+vEMovsT8CbwCuAjYh2U6DGVVG2M5ikU5Cdzk0XjOcr543jjQ0hHl+ymV+9HuCXrwc4d0Ix158xigtPKCEt1c5qjIlFLAkmS1Vv9T2SfiJY20B+lhW5TGapKcIFk0q4YFIJ2/Ye5MnlW1i4fDNzH3uXopx0Pj1tOFedNoLJpbHUfDWm/4olwfxZRK5Q1ed8j6YfCNTUM7bIilz2FsPyB/LtSyby9QvH89q6EE+9u4UF72zi4bc2Ul6ay1WnjWDWtGEU5WQkOlRjkk6HxS7ds2CygUa8h5AJoDGW7E9qiSh2aUUue7/dDYd59oNtPL2impXVdaSlCOdPKuYzp4zgwhNKGJiemugQjfFVrMUuY5lFNig+IZl9rsil1SDr3Qqy05kzo4w5M8pYv3M/T6+o5o/vbeWVtTVkpady8eQhfHJqKedNKiYjzZKN6b+iJhgROUFV/y4ip0Z6X1VX+BdW39Ra5NKqKPcdE4cM4vbLJ/O9y05gaXAXz67czvOrtrP4g20Mykzj0vKhfPLkUs4eX8QAmxxg+pn2zmBuAeYCP4vwngIX+hJRHxY8WuTSzmD6mtQUYcb4ImaML+KOWVN4u6qWP6/czourd/D0imryswZwafkQLi0fytkTisgcYGc2pu+LmmBUda77fkHPhdO3BUL1rsil3VPRlw1ITeH8SSWcP6mEOz9zIm+sr+XPK7fx/Ic7WFRZTVZ6KudNLObSKUO4cNIQ8rIGJDpkY3wRyywyRGQGH7/R8lGfYuqzgqEGK3LZz2SkpXJJ+RAuKR/C4aYWlgR38dKaHby0eifPr9pBWopwxtgCLi0fysXlQxiePzDRIRsTN7HMInsMGAe8z7EbLVVVv+FzbL7r6Vlkl977OqMKsnh4zuk9tk+TnFpalA+q9/LSmp28tHoHATc+N3FIDhdMKuG8ScVUjC6wP0ZMUorbLDKgAijXLjypSURmAj8HUoGHVfWuNu9nAI8CpwG7gGtVdZN773bgBryk9g1VfbG9PsW7seQ/gWvcNr9U1fs7G7NfmluUTbsOcP6kkkSHYpJASopwyqjBnDJqMLfOPIGqmnr+uq6G19bVMP/tjfz3G0FyMtI4e3wR508q5vxJJQzNy0x02MZ0SiwJZhXeA8a2d6ZjEUkFHgQuAaqB5SKyWFXXhDW7AdijquNFZDZwN3CtiJTjPdhsCjAMeEVEJrptovX5z8BI4ARVbRGRpPpN3lrk0p5iaSIZX5LD+JIcbjxnLPWNTfytqpbX1oV4fV0NL6zeAcDk0lzOnVDEWeOLOL2swO63MUkvlgRTBKwRkWV4N1sCoKpXdrDddKDKVWNGRBYCs4DwBDML+JFbfgp4wJ2JzAIWqmojsFFEqlx/tNPnV4HPqWqLi68mhs/WY1ofkzzWZpCZDuRkpHHplKFcOmUoqsr6nfW8tq6Gv4ad3aSnpnDq6HzOGlfEWROKmDo8z2qkmaQTS4L5URf7Hg5sCXtdDZwRrY2qNolIHVDo1i9ps+1wtxytz3F4Zz+fAUJ4l9U2dDH2uAvYFGXTBSLCpKGDmDR0EF85bxwHDjexfNMe/lZVy1tVtdzzynp+9vJ6BmWkccbYQs4aX8hZ44sYX5xDSoqVIzKJ1W6CcZe5/k1VL+5C35H+dbcdx4nWJtr6SH+itfaZARxS1QoR+UdgPnDOx4ISmYt3fw+jRo2KHLkPAiErcmm6Lys9jfMmFnPexGLAK1vzTmAXbwdqebuqllfW7gRgcNYATi8rYPqYAs4YU8jk0kF2hmN6XLsJRlWbReSAiOSpal0n+67GGxNpNQLYFqVNtYikAXnA7g62jba+GnjaLT8D/DpSUKo6D5gH3iyy2D9O9wRD9Vai38RdQXY6/zC1lH+YWgrAlt0HeCe4i+Ubd7Ns025eWuMlnOz0VE4rK+CMMV7SmToiz8rYGN/FconsEPChiLwMNLSujGGa8nJggoiMAbbiDdp/rk2bxcAc4B3gauBVVVURWQw8ISL34A3yTwCW4Z3ZROvzj3jVBeYD5wHrY/hsPSZY28D57q9OY/wysiCLkQVZfLbC+zts575DLNu4++jXT15cB3iPjp42Mp+K0YPdbLZ8qwht4i6WBPO/7qtT3JjKzcCLeFOK56vqahG5A6hU1cXAI8BjbhB/N17CwLVbhDd43wTcpKrNAJH6dLu8C3hcRL4N1AM3djZmv7QWubQBftPThuRm8qmTh/Gpk4cBsKfhMMs3eclm+abdzHsjSFOLdyI/qiCLU0blc6pLOJNLc61+mumWDm+07Mt66kbL97fs5dMPvs28L5zGpVOG+r4/Y2J16Egzq7bWsWLzHt7bvJcVm/ewc583WTQjLYWpI/K8M5yR+Zw8Mp/SvEx7lpGJ342WIjIB+C+gHDh6p5eqju1WhP3I0cck2xmMSTKZA1KpKCugoqzg6Lptew/y3ua9vLd5Dys27+E3b29iXnMLAEU56Zw4PI+TWr9G5DE015KOiSyWS2S/Bn4I3AtcAHyRyLO8TBTBWityaXqPYfkDGZY/8OjEgcamZtZu38+H1XtZWV3Hh1vreHNDLc3u0lpRTgYnDc/lpBH5nDQ8j6kj8hiSa1UHTGwJZqCq/kVERFU/An4kIm/iJR0Tg0BNA6OtyKXppTLSUpk2Mp9pI/OPrjt4uJm1O/bxYXUdK6vrWLW1jtfXb8DlHIoHZTC5NJfJpYMoL81lcmkuY4uybap0PxPTLDIRSQE2uAH2rUBSlWFJdsHaenvImOlTBqancuqowZw6avDRdQcON7F2+z6XcPaxdvs+5gdqOdLsZZ30tBQmDslh8tBcl3xyKS/NtccV9GGxJJhvAVnAN4D/wLtMNsfPoPqS5hZlU+0BLrAil6aPy0pP47TRBZw2+th4zuGmFgKhetZu3+e+9vPq32v4/bvVR9sMy8tkcmkuJ5QOYuKQQUwoGcS4kmy7T6cP6DDBqOpyAO8KmX7R/5D6luo9Bzjc3GJnMKZfSk9LOXq20kpVCe1vZI1LOK3J56/rQ0fHdVIEygqzmTAkx0s6QwYxcUibxNVSAAAUM0lEQVQOY4ty7FJzLxLLLLJP4N2vkgOMEpGTgX9R1a/5HVxfEHTP+bAaZMZ4RISS3ExKcjOPe3xFY1MzG2sbWL+zng0797N+53427Kzn5TU7j47tpKYIZYVZH0s6Y4qyrbp0EorlEtl9wGV4d92jqh+IyLm+RtWHWBVlY2KTkZbKCUNzOWFo7nHrDx1pJhhqYEONl3TW7/Quub2wegfht/ENzx/I2OJsxhZlM7Y4h3HFOYwtzmZobqYV/kyQmB6ZrKpb2sxzb47W1hzPilwa0z2ZA1IpH5ZL+bDIiSdYW+99D9UTCDXw1LvVNBw+9itq4IBUxhRle8mnOIdxxdneWU9xNjkZMf0KNF0Uy9HdIiIzABWRdLzB/rX+htV3BEP1dnnMGB9ESzyqSs3+RgKh1sTjJaGV1XU89+H2o5fbwLuHp6wwi1GFWYwuyKasKItRBVmUFWaTnzXAbiDtplgSzFfwHlE8HK9i8UuAjb/EKBBq4IJJVuTSmJ4iIgzJzWRIbiYzxhUd996hI81s3n3g6NnO5l0H2LSrgXcCu/jDiq3HtR2UmcbowixGF2Yz2iWdUYVZjC7MYsggu+wWi1hmkdUC14evE5Fv4Y3NmHbUHTxCbX0j40rsDMaYZJA5IJWJQ7zp0G0dOtLMlt0H+Mglnc27D7Bp1wFWb63jxVU7jhYFBa9O26gCL9mMGJzFiMED3Ze3nDfQzn4gxjGYCG7BEkyHgq0D/PYcGGOSXuaAVCa42WltNTW3sG3vIT7a3cCmXQfYvMv7vmX3AZYEd1Pf2HRc+0EZaQwPSzjhyWfk4CxyB6b1iwTU1QTT949MHLROUbYZZMb0bmmpKYxyYzXnTDj+PVWl7uARqvccpHrPAfe9dfkAS4K7OkxAw/MHUpqfSWneQIblZ1IyKJPUPnAJrqsJpv/W+O+EQKietBRhdKEVuTSmrxIR8rPSyc/yKk231ZUElJoiDBmUQWn+QErzMr0CpHmZlOYPZFiel4wKs9OT/iwoaoIRkf1ETiQCDPQtoj4kGGpgVEGWPbTJmH4slgS071AT2+sOsn3vIba1+b5qax0vrdnJ4aaW47ZLT0uhNC/TS0B5x86AhroJDkPyMijKzkjoZISoCUZVP34h0nSKV+TSLo8ZY6ITEfIGDiBv4ICP3WTaSlXZ3XCY7XWH2Lb3oPfdJaDtdQdZunE3O/YdOlpqp1VailAyKIMheZlHE89QtzxjXCElPj9Wwe4y8okVuTTGxIuIUJiTQWFORsSzIPB+59TWN7Kj7hA79h1i575Dxy2v37mfNzfUHr0c9+iXpluC6a1ai1zaTZbGmJ6QmnLs/p+T22lX39jEjrpDlOb5/1A4SzA+OVaDzKYoG2OSR05GGuN76N48X0efRWSmiKwTkSoRuS3C+xki8qR7f6mIlIW9d7tbv05ELutEn78QkXq/PlOsbIqyMaa/8y3BiEgq8CBwOVAOXCci5W2a3QDsUdXxwL3A3W7bcmA2MAWYCTwkIqkd9SkiFUA+SSAQamCwFbk0xvRjfp7BTAeqVDWoqoeBhcCsNm1mAQvc8lPAReJN7J4FLFTVRlXdCFS5/qL26ZLPT4Dv+fiZYhYI2QwyY0z/5meCGQ5sCXtd7dZFbKOqTUAdUNjOtu31eTOwWFW3txeUiMwVkUoRqQyFQp36QJ0RDDUwzsZfjDH9mJ8JJtLdPW1v3IzWplPrRWQYcA3wi46CUtV5qlqhqhXFxf5UOW4tcmlnMMaY/szPBFMNjAx7PQLYFq2NiKQBecDudraNtv4UYDxQJSKbgCwRqYrXB+ksK3JpjDH+JpjlwAQRGeMeVDYb99jlMIuBOW75auBVVVW3frabZTYGmAAsi9anqv6vqg5V1TJVLQMOuIkDCRFwM8isTL8xpj/z7T4YVW0SkZuBF4FUYL6qrhaRO4BKVV0MPAI85s42duMlDFy7RcAaoAm4SVWbASL16ddn6KqgK3I5qsCKXBpj+i9fb7RU1eeA59qs+0HY8iG8sZNI294J3BlLnxHaJPTUIRhqYFShFbk0xvRv9hvQB4FQPWOL7PKYMaZ/swQTZ03NLXy06wDjSmyA3xjTv1mCibPqPQe9Ipd2BmOM6ecswcRZsNaKXBpjDFiCibvWIpdWpt8Y099ZgomzQKiewVkDGGxFLo0x/ZwlmDgLhBrs7MUYY7AEE3fBUL2NvxhjDJZg4qruwBFq6w9bkUtjjMESTFwF3Awyu0RmjDGWYOLq2GOS7RKZMcZYgokjK3JpjDHHWIKJo0Co3opcGmOMY78J4yhoU5SNMeYoSzBx0tTcwqZdDTb+YowxjiWYOKnec5AjzWpFLo0xxrEEEyetRS6tTL8xxngswcRJoMZNUbYzGGOMASzBxE2wtp6C7HQrcmmMMY6vCUZEZorIOhGpEpHbIryfISJPuveXikhZ2Hu3u/XrROSyjvoUkcfd+lUiMl9EBvj52doK1DQwtsgujxljTCvfEoyIpAIPApcD5cB1IlLeptkNwB5VHQ/cC9ztti0HZgNTgJnAQyKS2kGfjwMnACcBA4Eb/fpskQRrrcilMcaE8/MMZjpQpapBVT0MLARmtWkzC1jglp8CLhIRcesXqmqjqm4Eqlx/UftU1efUAZYBI3z8bMdpLXJp98AYY8wxfiaY4cCWsNfVbl3ENqraBNQBhe1s22Gf7tLYF4AXuv0JYhQ4+phkSzDGGNPKzwQjEdZpjG06uz7cQ8AbqvpmxKBE5opIpYhUhkKhSE067dhjku0SmTHGtPIzwVQDI8NejwC2RWsjImlAHrC7nW3b7VNEfggUA7dEC0pV56lqhapWFBcXd/IjRRZwRS5HWpFLY4w5ys8EsxyYICJjRCQdb9B+cZs2i4E5bvlq4FU3hrIYmO1mmY0BJuCNq0TtU0RuBC4DrlPVFh8/18cEQ/WMtiKXxhhznDS/OlbVJhG5GXgRSAXmq+pqEbkDqFTVxcAjwGMiUoV35jLbbbtaRBYBa4Am4CZVbQaI1Kfb5a+Aj4B3vHkC/EFV7/Dr84ULhBps/MUYY9rwLcGAN7MLeK7Nuh+ELR8Cromy7Z3AnbH06db7+lmiaWpu4aNdDVw0uSQRuzfGmKRl13S66WiRSzuDMcaY41iC6aZAyBW5tBlkxhhzHEsw3dQ6RdmKXBpjzPEswXRTIGRFLo0xJhJLMN0UDFmRS2OMicQSTDcFQvU2wG+MMRFYgumGugNH2NVw2KooG2NMBJZguqG1yKWdwRhjzMdZgumGQE1rFWU7gzHGmLYswXRDsLaBAalW5NIYYyKxBNMNgZp6RhVYkUtjjInEfjN2Q7DWilwaY0w0lmC6qLXIpQ3wG2NMZJZgumiLK3JpA/zGGBOZJZguCoZsirIxxrTHEkwXWRVlY4xpnyWYLgqGGijMTic/y4pcGmNMJJZguigQqrfxF2OMaYclmC7yqijb+IsxxkTja4IRkZkisk5EqkTktgjvZ4jIk+79pSJSFvbe7W79OhG5rKM+RWSM62OD69O3a1d7DxxmV8NhxpXYGYwxxkTjW4IRkVTgQeByoBy4TkTK2zS7AdijquOBe4G73bblwGxgCjATeEhEUjvo827gXlWdAOxxffsiYE+xNMaYDvl5BjMdqFLVoKoeBhYCs9q0mQUscMtPAReJiLj1C1W1UVU3AlWuv4h9um0udH3g+vy0Xx/s6BTlEkswxhgTjZ8JZjiwJex1tVsXsY2qNgF1QGE720ZbXwjsdX1E21fcBEKuyOXggX7twhhjej0/E4xEWKcxtonX+o8HJTJXRCpFpDIUCkVq0qGywiw+c8pw0qzIpTHGROXnb8hqYGTY6xHAtmhtRCQNyAN2t7NttPW1QL7rI9q+AFDVeapaoaoVxcXFXfhYMHv6KH589cld2tYYY/oLPxPMcmCCm92Vjjdov7hNm8XAHLd8NfCqqqpbP9vNMhsDTACWRevTbfOa6wPX5598/GzGGGM6kNZxk65R1SYRuRl4EUgF5qvqahG5A6hU1cXAI8BjIlKFd+Yy2227WkQWAWuAJuAmVW0GiNSn2+WtwEIR+U/gPde3McaYBBHvj//+qaKiQisrKxMdhjHG9Coi8q6qVnTUzkapjTHG+MISjDHGGF9YgjHGGOMLSzDGGGN8YQnGGGOML/r1LDIRCQEfdXHzIrwbPJONxdU5FlfnWFydk6xxQfdiG62qHd6p3q8TTHeISGUs0/R6msXVORZX51hcnZOscUHPxGaXyIwxxvjCEowxxhhfWILpunmJDiAKi6tzLK7Osbg6J1njgh6IzcZgjDHG+MLOYIwxxvjCEkwXiMhMEVknIlUiclsP7G+TiHwoIu+LSKVbVyAiL4vIBvd9sFsvInK/i22liJwa1s8c136DiMyJtr8OYpkvIjUisipsXdxiEZHT3GetcttGephcrHH9SES2uuP2vohcEfbe7W4f60TksrD1EX+27hERS128T7rHRXQU00gReU1E1orIahH5ZjIcr3biSujxcttlisgyEfnAxfbv7fUn3iM9nnT7XyoiZV2NuYtx/UZENoYds2lufU/+208VkfdE5M/JcKyOo6r21YkvvMcEBICxQDrwAVDu8z43AUVt1v0YuM0t3wbc7ZavAJ7He8rnmcBSt74ACLrvg93y4C7Eci5wKrDKj1jwnvvzCbfN88Dl3YjrR8B3IrQtdz+3DGCM+3mmtvezBRYBs93yr4CvxhBTKXCqWx4ErHf7TujxaieuhB4v11aAHLc8AFjqjkXE/oCvAb9yy7OBJ7sacxfj+g1wdYT2Pflv/xbgCeDP7R37njpW4V92BtN504EqVQ2q6mFgITArAXHMAha45QXAp8PWP6qeJXhP+iwFLgNeVtXdqroHeBmY2dmdquobeM/uiXss7r1cVX1HvX/5j4b11ZW4opkFLFTVRlXdCFTh/Vwj/mzdX5IXAk9F+IztxbRdVVe45f3AWmA4CT5e7cQVTY8cLxePqmq9eznAfWk7/YUfy6eAi9z+OxVzN+KKpkd+liIyAvgH4GH3ur1j3yPHKpwlmM4bDmwJe11N+/8540GBl0TkXRGZ69YNUdXt4P3CAEo6iM/PuOMVy3C3HM8Yb3aXKOaLuxTVhbgKgb2q2tTVuNzliFPw/vJNmuPVJi5IguPlLvm8D9Tg/QIOtNPf0Rjc+3Vu/3H/f9A2LlVtPWZ3umN2r4hktI0rxv139Wd5H/A9oMW9bu/Y99ixamUJpvMiXRf1eyreWap6KnA5cJOInNtO22jxJSLuzsYS7xh/CYwDpgHbgZ8lIi4RyQGeBr6lqvvaa5rguJLieKlqs6pOA0bg/RU9uZ3+eiy2tnGJyInA7cAJwOl4l71u7am4ROSTQI2qvhu+up1+evz/oyWYzqsGRoa9HgFs83OHqrrNfa8BnsH7T7fTnVbjvtd0EJ+fcccrlmq3HJcYVXWn+6XQAvwP3nHrSly1eJc40tqs75CIDMD7Jf64qv7BrU748YoUVzIcr3Cquhf4K94YRrT+jsbg3s/Du1Tq2/+DsLhmusuNqqqNwK/p+jHrys/yLOBKEdmEd/nqQrwzmqQ5Vr4NTPfVLyANb2BuDMcGvqb4uL9sYFDY8t/wxk5+wvEDxT92y//A8YOLy9z6AmAj3sDiYLdc0MWYyjh+MD1usQDLXdvWgc4ruhFXadjyt/GuMwNM4fhBzSDegGbUny3we44fOP1aDPEI3rX0+9qsT+jxaieuhB4v17YYyHfLA4E3gU9G6w+4ieMHrhd1NeYuxlUadkzvA+5K0L/98zk2yJ/QY3VcXF35BdPfv/BmiKzHuzb8fZ/3Ndb9YD8AVrfuD+/a6V+ADe576z9SAR50sX0IVIT19SW8Abwq4ItdjOd3eJdPjuD9hXNDPGMBKoBVbpsHcDcDdzGux9x+VwKLOf4X6PfdPtYRNlsn2s/W/RyWuXh/D2TEENPZeJcUVgLvu68rEn282okrocfLbTcVeM/FsAr4QXv9AZnudZV7f2xXY+5iXK+6Y7YK+C3HZpr12L99t+35HEswCT1W4V92J78xxhhf2BiMMcYYX1iCMcYY4wtLMMYYY3xhCcYYY4wvLMEYY4zxhSUYYzpJRArDqufukOMrEMdaNfjXIjKpE/ssFZHnXDXfNSKy2K0fKyKzu/pZjPGTTVM2phtE5EdAvar+tM16wfv/1RJxw87v5xFghao+6F5PVdWVInIxcLOqxlRM0pieZGcwxsSJiIwXkVUi8itgBVAqIvNEpFK8Z4j8IKztWyIyTUTSRGSviNzlzk7eEZGSCN2XElYMUVVXusW7gAvc2dM3XH/3iPfskpUicqPb38XiPQPmj+4M6EGXBI3xjSUYY+KrHHhEVU9R1a14JWEqgJOBS0SkPMI2ecDrqnoy8A7end5tPQAsEJFXReRfW2uZ4ZWaeU1Vp6nq/cBcvAKI0/EKMN4kIqNc2zOAbwEn4RWQTMRjJkw/YgnGmPgKqOrysNfXicgKvDOayXgJqK2Dqvq8W34Xr6bacVT1ObxKx4+4Pt4TkcIIfV0KfNGVlV8K5AMT3HtLVHWTqjbjFUc8u7MfzpjOSOu4iTGmExpaF0RkAvBNYLqq7hWR3+LVg2rrcNhyM1H+X6rqLuBx4HEReQEvQTS0aSZ4xQ3/ctxKb6ym7YCrDcAaX9kZjDH+yQX2A/vCnmbYJSJykYgMdMu5eBVuN7v+B4U1fRH4Wmu5dhGZ1LodcKaIjBKRVOCzwFtdjceYWNgZjDH+WQGswauQGwTe7kZfpwMPiMgRvD8Mf6mq77lp0aki8gHe5bMHgVHA+24Mv4ZjYy1/w3uI2BS855ks7kY8xnTIpikb0w/YdGaTCHaJzBhjjC/sDMYYY4wv7AzGGGOMLyzBGGOM8YUlGGOMMb6wBGOMMcYXlmCMMcb4whKMMcYYX/x/1YcnkoXbCt0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "temp_learning_rate_schedule = CustomSchedule(d_model)\n",
    "\n",
    "plt.plot(temp_learning_rate_schedule(tf.range(40000, dtype=tf.float32)))\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.xlabel(\"Train Step\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YgkDE7hzo8r5"
   },
   "source": [
    "## 损失函数与指标（Loss and metrics）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oxGJtoDuYIHL"
   },
   "source": [
    "由于目标序列是填充（padded）过的，因此在计算损失函数时，应用填充遮挡非常重要。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MlhsJMm0TW_B"
   },
   "outputs": [],
   "source": [
    "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "    from_logits=True, reduction='none')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "67oqVHiT0Eiu"
   },
   "outputs": [],
   "source": [
    "def loss_function(real, pred):\n",
    "    mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
    "    loss_ = loss_object(real, pred)\n",
    "\n",
    "    mask = tf.cast(mask, dtype=loss_.dtype)\n",
    "    loss_ *= mask\n",
    "\n",
    "    return tf.reduce_mean(loss_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "phlyxMnm-Tpx"
   },
   "outputs": [],
   "source": [
    "train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
    "train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(\n",
    "    name='train_accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aeHumfr7zmMa"
   },
   "source": [
    "## 训练与检查点（Training and checkpointing）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UiysUa--4tOU"
   },
   "outputs": [],
   "source": [
    "transformer = Transformer(num_layers, d_model, num_heads, dff,\n",
    "                          input_vocab_size, target_vocab_size, \n",
    "                          pe_input=input_vocab_size, \n",
    "                          pe_target=target_vocab_size,\n",
    "                          rate=dropout_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZOJUSB1T8GjM"
   },
   "outputs": [],
   "source": [
    "def create_masks(inp, tar):\n",
    "    # 编码器填充遮挡\n",
    "    enc_padding_mask = create_padding_mask(inp)\n",
    "\n",
    "    # 在解码器的第二个注意力模块使用。\n",
    "    # 该填充遮挡用于遮挡编码器的输出。\n",
    "    dec_padding_mask = create_padding_mask(inp)\n",
    "\n",
    "    # 在解码器的第一个注意力模块使用。\n",
    "    # 用于填充（pad）和遮挡（mask）解码器获取到的输入的后续标记（future tokens）。\n",
    "    look_ahead_mask = create_look_ahead_mask(tf.shape(tar)[1])\n",
    "    dec_target_padding_mask = create_padding_mask(tar)\n",
    "    combined_mask = tf.maximum(dec_target_padding_mask, look_ahead_mask)\n",
    "\n",
    "    return enc_padding_mask, combined_mask, dec_padding_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Fzuf06YZp66w"
   },
   "source": [
    "创建检查点的路径和检查点管理器（manager）。这将用于在每 `n` 个周期（epochs）保存检查点。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hNhuYfllndLZ"
   },
   "outputs": [],
   "source": [
    "checkpoint_path = \"./checkpoints/train\"\n",
    "\n",
    "ckpt = tf.train.Checkpoint(transformer=transformer, optimizer=optimizer)\n",
    "\n",
    "ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)\n",
    "\n",
    "# 如果检查点存在，则恢复最新的检查点。\n",
    "if ckpt_manager.latest_checkpoint:\n",
    "    ckpt.restore(ckpt_manager.latest_checkpoint)\n",
    "    print('Latest checkpoint restored!!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0Di_Yaa1gf9r"
   },
   "source": [
    "目标（target）被分成了 tar_inp 和 tar_real。tar_inp 作为输入传递到解码器。`tar_real` 是位移了 1 的同一个输入：在 `tar_inp` 中的每个位置，`tar_real` 包含了应该被预测到的下一个标记（token）。\n",
    "\n",
    "例如，`sentence` = \"SOS A lion in the jungle is sleeping EOS\"\n",
    "\n",
    "`tar_inp` =  \"SOS A lion in the jungle is sleeping\"\n",
    "\n",
    "`tar_real` = \"A lion in the jungle is sleeping EOS\"\n",
    "\n",
    "Transformer 是一个自回归（auto-regressive）模型：它一次作一个部分的预测，然后使用到目前为止的自身的输出来决定下一步要做什么。\n",
    "\n",
    "在训练过程中，本示例使用了 teacher-forcing 的方法（就像[文本生成教程](./text_generation.ipynb)中一样）。无论模型在当前时间步骤下预测出什么，teacher-forcing 方法都会将真实的输出传递到下一个时间步骤上。\n",
    "\n",
    "当 transformer 预测每个词时，*自注意力（self-attention）*功能使它能够查看输入序列中前面的单词，从而更好地预测下一个单词。\n",
    "\n",
    "为了防止模型在期望的输出上达到峰值，模型使用了前瞻遮挡（look-ahead mask）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LKpoA6q1sJFj"
   },
   "outputs": [],
   "source": [
    "EPOCHS = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iJwmp9OE29oj"
   },
   "outputs": [],
   "source": [
    "# 该 @tf.function 将追踪-编译 train_step 到 TF 图中，以便更快地\n",
    "# 执行。该函数专用于参数张量的精确形状。为了避免由于可变序列长度或可变\n",
    "# 批次大小（最后一批次较小）导致的再追踪，使用 input_signature 指定\n",
    "# 更多的通用形状。\n",
    "\n",
    "train_step_signature = [\n",
    "    tf.TensorSpec(shape=(None, None), dtype=tf.int64),\n",
    "    tf.TensorSpec(shape=(None, None), dtype=tf.int64),\n",
    "]\n",
    "\n",
    "\n",
    "@tf.function(input_signature=train_step_signature)\n",
    "def train_step(inp, tar):\n",
    "    tar_inp = tar[:, :-1]\n",
    "    tar_real = tar[:, 1:]\n",
    "\n",
    "    enc_padding_mask, combined_mask, dec_padding_mask = create_masks(\n",
    "        inp, tar_inp)\n",
    "\n",
    "    with tf.GradientTape() as tape:\n",
    "        predictions, _ = transformer(inp, tar_inp, True, enc_padding_mask,\n",
    "                                     combined_mask, dec_padding_mask)\n",
    "        loss = loss_function(tar_real, predictions)\n",
    "\n",
    "    gradients = tape.gradient(loss, transformer.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(gradients, transformer.trainable_variables))\n",
    "\n",
    "    train_loss(loss)\n",
    "    train_accuracy(tar_real, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qM2PDWGDJ_8V"
   },
   "source": [
    "葡萄牙语作为输入语言，英语为目标语言。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bbvmaKNiznHZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 0 Loss 4.3110 Accuracy 0.0000\n",
      "Epoch 1 Batch 50 Loss 4.2296 Accuracy 0.0007\n",
      "Epoch 1 Batch 100 Loss 4.1944 Accuracy 0.0114\n",
      "Epoch 1 Batch 150 Loss 4.1565 Accuracy 0.0167\n",
      "Epoch 1 Batch 200 Loss 4.0987 Accuracy 0.0194\n",
      "Epoch 1 Batch 250 Loss 4.0298 Accuracy 0.0222\n",
      "Epoch 1 Batch 300 Loss 3.9390 Accuracy 0.0262\n",
      "Epoch 1 Batch 350 Loss 3.8438 Accuracy 0.0296\n",
      "Epoch 1 Batch 400 Loss 3.7598 Accuracy 0.0322\n",
      "Epoch 1 Batch 450 Loss 3.6778 Accuracy 0.0348\n",
      "Epoch 1 Batch 500 Loss 3.6052 Accuracy 0.0377\n",
      "Epoch 1 Batch 550 Loss 3.5494 Accuracy 0.0410\n",
      "Epoch 1 Batch 600 Loss 3.4873 Accuracy 0.0446\n",
      "Epoch 1 Batch 650 Loss 3.4307 Accuracy 0.0480\n",
      "Epoch 1 Batch 700 Loss 3.3753 Accuracy 0.0515\n",
      "Epoch 1 Loss 3.3741 Accuracy 0.0516\n",
      "Time taken for 1 epoch: 899.3856389522552 secs\n",
      "\n",
      "Epoch 2 Batch 0 Loss 2.5529 Accuracy 0.1032\n",
      "Epoch 2 Batch 50 Loss 2.5547 Accuracy 0.1021\n",
      "Epoch 2 Batch 100 Loss 2.5329 Accuracy 0.1045\n",
      "Epoch 2 Batch 150 Loss 2.5108 Accuracy 0.1065\n",
      "Epoch 2 Batch 200 Loss 2.4919 Accuracy 0.1088\n",
      "Epoch 2 Batch 250 Loss 2.4776 Accuracy 0.1110\n",
      "Epoch 2 Batch 300 Loss 2.4588 Accuracy 0.1129\n",
      "Epoch 2 Batch 350 Loss 2.4421 Accuracy 0.1149\n",
      "Epoch 2 Batch 400 Loss 2.4294 Accuracy 0.1166\n",
      "Epoch 2 Batch 450 Loss 2.4194 Accuracy 0.1180\n",
      "Epoch 2 Batch 500 Loss 2.4074 Accuracy 0.1195\n",
      "Epoch 2 Batch 550 Loss 2.3948 Accuracy 0.1208\n",
      "Epoch 2 Batch 600 Loss 2.3820 Accuracy 0.1221\n",
      "Epoch 2 Batch 650 Loss 2.3704 Accuracy 0.1233\n",
      "Epoch 2 Batch 700 Loss 2.3596 Accuracy 0.1245\n",
      "Epoch 2 Loss 2.3593 Accuracy 0.1245\n",
      "Time taken for 1 epoch: 786.3909301757812 secs\n",
      "\n",
      "Epoch 3 Batch 0 Loss 1.8300 Accuracy 0.1326\n",
      "Epoch 3 Batch 50 Loss 2.1932 Accuracy 0.1455\n",
      "Epoch 3 Batch 100 Loss 2.1564 Accuracy 0.1435\n",
      "Epoch 3 Batch 150 Loss 2.1438 Accuracy 0.1443\n",
      "Epoch 3 Batch 200 Loss 2.1482 Accuracy 0.1451\n",
      "Epoch 3 Batch 250 Loss 2.1524 Accuracy 0.1461\n",
      "Epoch 3 Batch 300 Loss 2.1532 Accuracy 0.1471\n",
      "Epoch 3 Batch 350 Loss 2.1445 Accuracy 0.1472\n",
      "Epoch 3 Batch 400 Loss 2.1337 Accuracy 0.1478\n",
      "Epoch 3 Batch 450 Loss 2.1269 Accuracy 0.1487\n",
      "Epoch 3 Batch 500 Loss 2.1194 Accuracy 0.1496\n",
      "Epoch 3 Batch 550 Loss 2.1112 Accuracy 0.1505\n",
      "Epoch 3 Batch 600 Loss 2.1041 Accuracy 0.1513\n",
      "Epoch 3 Batch 650 Loss 2.0956 Accuracy 0.1521\n",
      "Epoch 3 Batch 700 Loss 2.0879 Accuracy 0.1529\n",
      "Epoch 3 Loss 2.0882 Accuracy 0.1530\n",
      "Time taken for 1 epoch: 830.8191289901733 secs\n",
      "\n",
      "Epoch 4 Batch 0 Loss 2.1082 Accuracy 0.1759\n",
      "Epoch 4 Batch 50 Loss 1.9278 Accuracy 0.1698\n",
      "Epoch 4 Batch 100 Loss 1.9418 Accuracy 0.1702\n",
      "Epoch 4 Batch 150 Loss 1.9201 Accuracy 0.1706\n",
      "Epoch 4 Batch 200 Loss 1.9132 Accuracy 0.1721\n",
      "Epoch 4 Batch 250 Loss 1.9105 Accuracy 0.1732\n",
      "Epoch 4 Batch 300 Loss 1.8969 Accuracy 0.1739\n",
      "Epoch 4 Batch 350 Loss 1.8938 Accuracy 0.1751\n",
      "Epoch 4 Batch 400 Loss 1.8888 Accuracy 0.1760\n",
      "Epoch 4 Batch 450 Loss 1.8823 Accuracy 0.1771\n",
      "Epoch 4 Batch 500 Loss 1.8738 Accuracy 0.1779\n",
      "Epoch 4 Batch 550 Loss 1.8667 Accuracy 0.1788\n",
      "Epoch 4 Batch 600 Loss 1.8633 Accuracy 0.1798\n",
      "Epoch 4 Batch 650 Loss 1.8584 Accuracy 0.1808\n",
      "Epoch 4 Batch 700 Loss 1.8498 Accuracy 0.1817\n",
      "Epoch 4 Loss 1.8497 Accuracy 0.1817\n",
      "Time taken for 1 epoch: 722.8582880496979 secs\n",
      "\n",
      "Epoch 5 Batch 0 Loss 1.8352 Accuracy 0.2027\n",
      "Epoch 5 Batch 50 Loss 1.6640 Accuracy 0.1972\n",
      "Epoch 5 Batch 100 Loss 1.6703 Accuracy 0.1992\n",
      "Epoch 5 Batch 150 Loss 1.6738 Accuracy 0.1998\n",
      "Epoch 5 Batch 200 Loss 1.6726 Accuracy 0.2007\n",
      "Epoch 5 Batch 250 Loss 1.6759 Accuracy 0.2019\n",
      "Epoch 5 Batch 300 Loss 1.6653 Accuracy 0.2022\n",
      "Epoch 5 Batch 350 Loss 1.6617 Accuracy 0.2030\n",
      "Epoch 5 Batch 400 Loss 1.6610 Accuracy 0.2041\n",
      "Epoch 5 Batch 450 Loss 1.6576 Accuracy 0.2049\n",
      "Epoch 5 Batch 500 Loss 1.6515 Accuracy 0.2055\n",
      "Epoch 5 Batch 550 Loss 1.6436 Accuracy 0.2058\n",
      "Epoch 5 Batch 600 Loss 1.6387 Accuracy 0.2063\n",
      "Epoch 5 Batch 650 Loss 1.6351 Accuracy 0.2067\n",
      "Epoch 5 Batch 700 Loss 1.6322 Accuracy 0.2073\n",
      "Saving checkpoint for epoch 5 at ./checkpoints/train/ckpt-1\n",
      "Epoch 5 Loss 1.6320 Accuracy 0.2073\n",
      "Time taken for 1 epoch: 736.1837697029114 secs\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(EPOCHS):\n",
    "    start = time.time()\n",
    "\n",
    "    train_loss.reset_states()\n",
    "    train_accuracy.reset_states()\n",
    "\n",
    "    # inp -> portuguese, tar -> english\n",
    "    for (batch, (inp, tar)) in enumerate(train_dataset):\n",
    "        train_step(inp, tar)\n",
    "\n",
    "        if batch % 50 == 0:\n",
    "            print('Epoch {} Batch {} Loss {:.4f} Accuracy {:.4f}'.format(\n",
    "                epoch + 1, batch, train_loss.result(),\n",
    "                train_accuracy.result()))\n",
    "\n",
    "    if (epoch + 1) % 5 == 0:\n",
    "        ckpt_save_path = ckpt_manager.save()\n",
    "        print('Saving checkpoint for epoch {} at {}'.format(\n",
    "            epoch + 1, ckpt_save_path))\n",
    "\n",
    "    print('Epoch {} Loss {:.4f} Accuracy {:.4f}'.format(\n",
    "        epoch + 1, train_loss.result(), train_accuracy.result()))\n",
    "\n",
    "    print('Time taken for 1 epoch: {} secs\\n'.format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QfcsSWswSdGV"
   },
   "source": [
    "## 评估（Evaluate）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y6APsFrgImLW"
   },
   "source": [
    "以下步骤用于评估：\n",
    "\n",
    "* 用葡萄牙语分词器（`tokenizer_pt`）编码输入语句。此外，添加开始和结束标记，这样输入就与模型训练的内容相同。这是编码器输入。\n",
    "* 解码器输入为 `start token == tokenizer_en.vocab_size`。\n",
    "* 计算填充遮挡和前瞻遮挡。\n",
    "* `解码器`通过查看`编码器输出`和它自身的输出（自注意力）给出预测。\n",
    "* 选择最后一个词并计算它的 argmax。\n",
    "* 将预测的词连接到解码器输入，然后传递给解码器。\n",
    "* 在这种方法中，解码器根据它预测的之前的词预测下一个。\n",
    "\n",
    "Note：这里使用的模型具有较小的能力以保持相对较快，因此预测可能不太正确。要复现论文中的结果，请使用全部数据集，并通过修改上述超参数来使用基础 transformer 模型或者 transformer XL。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5buvMlnvyrFm"
   },
   "outputs": [],
   "source": [
    "def evaluate(inp_sentence):\n",
    "    start_token = [tokenizer_pt.vocab_size]\n",
    "    end_token = [tokenizer_pt.vocab_size + 1]\n",
    "\n",
    "    # 输入语句是葡萄牙语，增加开始和结束标记\n",
    "    inp_sentence = start_token + tokenizer_pt.encode(inp_sentence) + end_token\n",
    "    encoder_input = tf.expand_dims(inp_sentence, 0)\n",
    "\n",
    "    # 因为目标是英语，输入 transformer 的第一个词应该是\n",
    "    # 英语的开始标记。\n",
    "    decoder_input = [tokenizer_en.vocab_size]\n",
    "    output = tf.expand_dims(decoder_input, 0)\n",
    "\n",
    "    for i in range(MAX_LENGTH):\n",
    "        enc_padding_mask, combined_mask, dec_padding_mask = create_masks(\n",
    "            encoder_input, output)\n",
    "\n",
    "        # predictions.shape == (batch_size, seq_len, vocab_size)\n",
    "        predictions, attention_weights = transformer(encoder_input, output,\n",
    "                                                     False, enc_padding_mask,\n",
    "                                                     combined_mask,\n",
    "                                                     dec_padding_mask)\n",
    "\n",
    "        # 从 seq_len 维度选择最后一个词\n",
    "        predictions = predictions[:, -1:, :]  # (batch_size, 1, vocab_size)\n",
    "\n",
    "        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
    "\n",
    "        # 如果 predicted_id 等于结束标记，就返回结果\n",
    "        if predicted_id == tokenizer_en.vocab_size + 1:\n",
    "            return tf.squeeze(output, axis=0), attention_weights\n",
    "\n",
    "        # 连接 predicted_id 与输出，作为解码器的输入传递到解码器。\n",
    "        output = tf.concat([output, predicted_id], axis=-1)\n",
    "\n",
    "    return tf.squeeze(output, axis=0), attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CN-BV43FMBej"
   },
   "outputs": [],
   "source": [
    "def plot_attention_weights(attention, sentence, result, layer):\n",
    "    fig = plt.figure(figsize=(16, 8))\n",
    "\n",
    "    sentence = tokenizer_pt.encode(sentence)\n",
    "\n",
    "    attention = tf.squeeze(attention[layer], axis=0)\n",
    "\n",
    "    for head in range(attention.shape[0]):\n",
    "        ax = fig.add_subplot(2, 4, head + 1)\n",
    "\n",
    "        # 画出注意力权重\n",
    "        ax.matshow(attention[head][:-1, :], cmap='viridis')\n",
    "\n",
    "        fontdict = {'fontsize': 10}\n",
    "\n",
    "        ax.set_xticks(range(len(sentence) + 2))\n",
    "        ax.set_yticks(range(len(result)))\n",
    "\n",
    "        ax.set_ylim(len(result) - 1.5, -0.5)\n",
    "\n",
    "        ax.set_xticklabels(['<start>'] +\n",
    "                           [tokenizer_pt.decode([i])\n",
    "                            for i in sentence] + ['<end>'],\n",
    "                           fontdict=fontdict,\n",
    "                           rotation=90)\n",
    "\n",
    "        ax.set_yticklabels([\n",
    "            tokenizer_en.decode([i])\n",
    "            for i in result if i < tokenizer_en.vocab_size\n",
    "        ],\n",
    "                           fontdict=fontdict)\n",
    "\n",
    "        ax.set_xlabel('Head {}'.format(head + 1))\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lU2_yG_vBGza"
   },
   "outputs": [],
   "source": [
    "def translate(sentence, plot=''):\n",
    "    result, attention_weights = evaluate(sentence)\n",
    "\n",
    "    predicted_sentence = tokenizer_en.decode(\n",
    "        [i for i in result if i < tokenizer_en.vocab_size])\n",
    "\n",
    "    print('Input: {}'.format(sentence))\n",
    "    print('Predicted translation: {}'.format(predicted_sentence))\n",
    "\n",
    "    if plot:\n",
    "        plot_attention_weights(attention_weights, sentence, result, plot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YsxrAlvFG8SZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: este é um problema que temos que resolver.\n",
      "Predicted translation: this is a problem that we have to solve this .\n",
      "Real translation: this is a problem we have to solve .\n"
     ]
    }
   ],
   "source": [
    "translate(\"este é um problema que temos que resolver.\")\n",
    "print (\"Real translation: this is a problem we have to solve .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7EH5y_aqI4t1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: os meus vizinhos ouviram sobre esta ideia.\n",
      "Predicted translation: my children have to have this idea about this .\n",
      "Real translation: and my neighboring homes heard about this idea .\n"
     ]
    }
   ],
   "source": [
    "translate(\"os meus vizinhos ouviram sobre esta ideia.\")\n",
    "print (\"Real translation: and my neighboring homes heard about this idea .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "J-hVCTSUMlkb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: vou então muito rapidamente partilhar convosco algumas histórias de algumas coisas mágicas que aconteceram.\n",
      "Predicted translation: so i 'll give you very little lot of some lot of some of things that happens .\n",
      "Real translation: so i 'll just share with you some stories very quickly of some magical things that have happened .\n"
     ]
    }
   ],
   "source": [
    "translate(\"vou então muito rapidamente partilhar convosco algumas histórias de algumas coisas mágicas que aconteceram.\")\n",
    "print (\"Real translation: so i 'll just share with you some stories very quickly of some magical things that have happened .\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_1MxkSZvz0jX"
   },
   "source": [
    "您可以为 `plot` 参数传递不同的层和解码器的注意力模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t-kFyiOLH0xg"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: este é o primeiro livro que eu fiz.\n",
      "Predicted translation: this is the first book that i did n't do .\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABDIAAAI4CAYAAACY4sl8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xm4JHV59//3PQsMDIOIgCCEVRYBgcGJCoLBJaJGnyeoxIiJougYcUNDjCREEyN5TMyTJ5HEBcGBqHFF8sMNd0BAQYRZQECNguKSKIqMrMOZ+/dH15Geceac7q6uqlNd79d19XW6q/uu7316qj/T851aIjORJEmSJElqg3lNNyBJkiRJkjQoJzIkSZIkSVJrOJEhSZIkSZJaw4kMSZIkSZLUGk5kSJIkSZKk1nAiQ5IkSZIktYYTGZIkSZIkqTWcyJAkSZIkSa3hRIYkSZIkSWoNJzIkSZIkSVJrLGi6AY0mIg4Fji4efiUzVzXZj6TJY85IqoNZI6kOZs1kcY+MFoqI1wAfAHYqbu+PiFc125WkSWLOSKqDWSOpDmbN5InMbLoHDSkiVgNHZOadxePFwFcz85BmO5M0KcwZSXUwayTVwayZPO6R0U4BTPU9niqWSdK4mDOS6mDWSKqDWTNhWn2OjIgI4ALgtMy8oel+arQCuDIiLige/z5wToP9SBOto1ljzkg1M2sAs0aqnFkDmDWt1+pDSyLiWHob4Icz80+b7qdOEXE4cBS9mcRLM/PahluSJlZXs8ackepl1pg1Uh3MGrNmErR9IuMjwHuBtwMHZub9DbdUuYiYB6zOzIOb7kXqiq5ljTkjNcOskVQHs0aToLXnyIiIHYCDMvMi4AvAcQ23VIvMXA+siojdm+5F6oIuZo05I9XPrJFUB7NGk6K1ExnAC4APFvdXACc12EvddgGuj4gvRsSF07emm+qSiDguIrZpug/VoqtZY87MAWZNp5g1Zk1jzJpOMWvMmsaMM2tae2hJRKwBnpqZPywerwKekZk/aLaz6kXE72xqeWZeUncvXRQR+wA3Aq/KzHc13Y+q1dWsMWeaZ9Z0i1mzIbOmPmZNt5g1GzJr6jPurGnlREZEbAc8NzPf3bfsd4GfedIWVS0izgASeEpmPrrpflQds0ZNMmu6w6xRk8ya7jBr1KRxZ00rDy3JzNuB6zZa9nlg62Y6qkdEXFb8XBsRd/Td1kbEHU331wURMR84Hvh74JcRcWjDLalCXcwac2ZuMGu6xawxa5pi1nSLWWPWNKWKrGnlREbhzAGXTYzMPKr4uSQzt+27LcnMbZvuryOeDlyRmWvpne35JQ33o+p1KmvMmTnDrOkes8asaYJZ0z1mjVnThLFnzYLSLdUsIo4AjgR2jIjX9T21LTC/ma7qFxFHAftm5ori7MNLMvN7TffVAScB/7e4fwHwloj408y8r8GeVAGzxpxpmFnTEWaNWdMws6YjzBqzpmFjz5o27pGxBbANvUmYJX23O4DnNNhXbSLiTcCfA6cVi7YA3t9cR91QHFe4XWZ+BSAz7wE+Bjyx0cZUlU5njTnTHLOmc8was6YRZk3nmDVmTSOqypq2nuxzPvDhzJz4D92mRMRKYClwTWYuLZatzsxDmu1MmixdzhpzRqqPWWPWSHUwa8yaSdK6Q0sAMnMqIrZvuo8G3ZeZGREJEBGLm25o0kXE4TM9n5nX1NWL6tPxrDFnGmDWdJNZY9bUzazpJrPGrKlblVnTyomMwrURcSHwUeDO6YWZ+fHmWqrNRyLi3cB2EfFS4MXAexruadJNH9O1CFgGrAICOAS4Ejiqob5Uva5mjTnTDLOmu8was6ZOZk13mTVmTZ0qy5pWHloCEBErNrE4M/PFtTfTgOKaz0+htyF8trh0kioWER8CzsjMNcXjg4FTM/PERhtTZbqcNeZMc8ya7jFrzJommDXdY9aYNU2oImtaO5EhiIht6durJjN/3mA7nRARKzPzsNmWSZPCnGmGWaOuMWuaYdaoa8yaZlSRNa09tCQiFtG7jMtB9HZVAaAjs4kvA94M3A2spzermMDeA9ZvAexXPLwpM9dV0eeEuiEizqZ3luME/gi4odmWVKWuZk3ZnCnWYdaMzqzpGLPGrGmIWdMxZo1Z05CxZ00bL7867X3AzsCxwCXAbsDaQQoj4qERcU5EfKZ4fGBEnFRZp+N3KnBQZu6ZmXtn5l6ZOegkxjHAt4F/A94BfCsiHl9dqxPnRcD1wGuAU4BvFss0ubqaNSPnDJg1Y2DWdI9ZY9Y0wazpHrPGrGnC2LOmtYeWRMS1mbl0+rI5EbGQ3rFOs16PtvjwrQD+MjMPjYgFwLWZ+ciq+x6HiLgIeFZm3jVC7TeAEzLzpuLxfsAHM/NRY25TmghdzZoyOVPUmzXSEMwas0aqg1lj1kyK1h5aAkzvynN7cbKQnwB7Dli7Q2Z+JCJOA8jM+yNiqoIeq3IacEVEXAncO70wM189QO3C6Q9gUfOtIsAGFhF7APtm5hciYitgQWYONJPbdhHxOOCvgT3Y8Pi6gWd01TpdzZoyOQNmTSlmTSeZNWZN7cyaTjJrzJraVZE1bZ7IOCsiHgycDlwIbAP81YC1d0bEQ+gdn0NEPBb4ZSVdVuPdwJeANfSO8RrG1RFxDr3dygCeD3xj0OLoXa5oObA9sA+93dHeBTxpyD7a6hzgtfTes7YEt8rpataUyRkwa8oya7rHrDFrmmDWdI9ZY9Y0YexZ0+ZDS/bKzO/NtmwztYcDZwIHA9cBOwLHZ+aqSpods4i4IjOPHLF2S+AV9K7ZG8ClwDsy894ZCx+oXwk8GrgyM5cWy9a0YZeycYiIKzPzMU33ofp0NWvK5ExRb9aUYNZ0j1kzcr1ZU4JZ0z1mzcj1Zk0JVWRNmycyrsnMwzda9o1BjlMqNsQpYH96G+JNwLxBN8SmRcQZwC3AJ9hw16gZLx8UEfOB8zLzj0qMfWVmPqbv+LoFwDWZecio62yTiHgrMB/4OBu+99c01pQq1dWsGTVnilqzpiSzpnvMGrOmCWZN95g1Zk0Tqsia1h1aEhEH0Ltc0IMi4ll9T21L3yWEZvHV4gN8fd96rwEO33zJnHJC8fO0vmWzXj4oM6ciYseI2CIz7xtx7Esi4i+ArSLid4GT6QVCV0zPJC7rW5bArCdIUruYNaPlDJg1Y2LWdIRZY9Y0zKzpCLPGrGnY2LOmdRMZ9GYAnwFsBzyzb/la4KUzFUbEzsCu9DagpfRmEqH3Ad56/K1WIzP3KlF+M3B5RFwI3Nm3zn8asP4N9K49vQZ4GfBp4OwS/bRKZj6h6R5Um05nTcmcAbOmFLOmU8yacm7GrBmZWdMpZk05N2PWjKyKrGnzoSVHZOZXh6x5IXAivZmgr/PAh3AtcG5mfnysTY5ZRDwxM7+00Szqrw3Sf0S8aTO1f1O2vy6IiIcCfwc8LDOfFhEHAkdk5jkNt6aKdC1rxpEzxXrMmhLMmu4xazZk1tTDrOkes2ZDZk09qsiaNk9k/APwFuBu4CLgUOCUzHz/ALXPzszzK25x7CLibzLzTRGxYhNPZ2a+eIB1LM3Ma0v08D2KMxVvNPhAl84pet9U/ay9zwXR4utnazRdy5px5EyxHrOmBLOme8yaDZg1NTFruses2UArsqbtOQPVZE0bDy2Z9pTMfH1EHAfcChwPfBmY9UMI7BYR29KbRXwPveO63pCZn6us2zEoPoTzgM9k5kdGXM0/RcQuwEeBD2Xm9bMVbKT/uKZF9N737Yeo/+RG9ccBPxq0OCKOBq7IzKm+ZYfXeFKqNl8/W6PpVNaMKWfArCnLrOkes2Y0Zk05Zk33mDWjaTJrSuUMTGjWZGYrb8D1xc/3AE8t7q8asHZV8fNYetdPPpTeWWMb/70G7P/SkvU7A68GLqd3nNbpJdd3WYnaecCXhnj9XcAlwEP7ltX2ZwdcDDxkekzgscAlTW8T3ir9M+9k1pTNmWIdZs3o/Zo1HbuZNaXWYdaM3q9Z07GbWVNqHXMia4bNmaJm4rKmzXtkfCIibqS3W9TJEbEjcM+AtdPHdT0dWJGZqyIiZiqYYz4fEacCH2bDk83Mevmg4nU/Ad4eEV8GXg+8kd4uZrOK3vWjp82jN7u4ZMC+N2VfYPchXn8T8Dbg4og4KTOv4IE/zzq8jl5w7xMRl9O7fvZzahxf9etq1pTKmeK1Zs3ozJruMWvMGrNGdTBr2p81w+YMTGDWtPYcGQAR8WDgjuxdEmcxsKTYwGarW0HvzLt70ZtJnA9cnANcP3kuKI6x2ljmYMdYPQJ4Lr0N5zbgQ8D5mfk/A4795b6H99M7g+8/ZuZNA9avpXeMVxQ/fwKclgMebxfFta8jYl96QfRe4MW50fWwq1Qc0/Xr62dn5rq6xlYzupg1ZXKmqDdrSjJruses+TWzxqxRhcyaX2tF1pTNmWIdE5c1rZzIiIitgX0zc1Xfst2Bqcz84QD184DDgO9m5u0R8RBg18xcXVnTc0REfA34IPDRzBzq2Kq5ICKuzcylxf3F9E4a86zMrHzvorLbndrHrBmdWVNqbLOmY8ya0Zk1pcY2azrGrBmdWVNq7Eqypq0TGQuBG4FDMvPOYtnngL/IzKsHqA/g+cDemfnm4o3cOTOvGnD8UvXFOg4Fji4efqX/D3aA2kXAycBR9GblvgK8KzMH3S1sZBHxupmez1mupdz33u2VmX87ynu3iXXunpnfH7V+iHFKbXdqny5nTZM5U4xv1pg1ndH2rGnrd5pifLPGrOmMJrOmy/9+KsYfOWuqyJliva3Omnlj6q9WxW4oF9DbvWd6RmfHId6IdwBHAM8rHq8F/m2mgoh4XETMH7V+o3W9BvgAsFNxe39EvGrQeuDfgYOAM4F/BQ4E3jfLmB8pfq6JiNV9tzURMcws6jLg5fR2K9sV+JNi/CUMdpzX9Ht3QvF4oPcuIl5f/DwzIt7efwNOHaL/kY1hu1PLdDxrhs6ZYkyzpiSzpnvanDVNfKcpxjVrSjJruqfurJlD32mg3VkzUs4UfU9u1mRNZyod9w04gN5MHMDpwKuHqJ0+W+q1fctmPFsvcCRw1qj1G61rNbC47/FiYPUQ9b8x1gD971L83GNTtyHG/hy94+imHy8BLqryvS9ec1vx8xTghRvf2rDdeWvnratZM0rOFK8xaxre7ry189bWrCmTM5sbx6wxa7zNzT/zYbf3ceVM8drOZk3JjJ7YrGntVUsy88aIICL2ozezd9QQ5euK2cEEiN7ZetfPMt4VEXHXqPUbCaD/urlTxbJBXRsRj83MrxXjP4beZYA2KzN/XPy8ZYhxNmV34L6+x/cBew5RP+p7998RsQfwIuAJQ4w3ViW3O7VQh7Nm6JwBs2ZczJruaXHW1P6dBsyacTFruqfOrJlD32mg3VlT5r2b2Kxp7URG4RzgbHqzcb8You7t9HZv2SkizqB39tnTZyvKzJVl6vusAK6MiAuKx79P73cZ1GOAF0TE9DFNuwM3RMSaXpt5yMYF8cDZbn/jqaJm2wHHfh9wVdF7AscB5w3R+6jv3TuBi4C9gf7dkKbP3jvQGYc3JyJ2zgHO1lwYdbtTe3Uxa4bOGTBrZmPWaBZtzJrav9OAWTOTIXMGzJouqi1r5sh3Gmh31pR57yY2a1p5ss9p0TsD6o+BZ2fmF4asPQB4Er0/xC9m5g011x9ObyYqgEsz89ohaveY6fkxzBjONv7hPHCinaF6L+pHfu8i4p2Z+fJhxhtwvZ/KzN8b8LUjb3dqpy5mTdM5U/Rg1pg1ndLWrGnzd5qih4nKmmFypni9WdMxTWVNl//9VPQwctaM4b2buKxp9USGJEmSJEnqllZetUSSJEmSJHWTExmSJEmSJKk1JmIiIyKWW9++sdteX3ZstU+bt9ey9W3uven6pntX+zS9zbR5e29zfZt7Vzu5vbezvs29j6N+2kRMZABl34wu17e596br/Qu/e9q8vZatb3PvTdc33bvap+ltps3be5vr29y72sntvZ31be59HPXA5ExkSJIkSZKkDmjFVUu2iC1zEYs3+/w67mUhW272+YcfcueM6//ZbVPs8JD5m33+v65bMmP9fXkPW8SizT6f69fPWD9b/7MpU9/k2G2vn632Hu7kvrw3Ru1N9SubNfsdcteM6//pbVPsOEPWfGv11jPWz+Xt3frmxjZr2meSs6bNn9Wm6+d672ZN+5TNmt0e+asZ13/7z9ez3fab/3/xH9704M0+d9/U3Wwxf6sZ15/3rdvsc3P981JlfZt7H6R+0KxZMHIHNVrEYh4TTxq5/oLPXFVq/Gfv/4RS9evvnHkiRZPpyvxi0y1oSGWz5rOfXVlq/GMfdlipenWTWdM+jWfNrktL1dOC/wTT+Jk17VM2a972ya+VGv+0o59Tqv7+H9xaql7tNGjWeGiJJEmSJElqDScyJEmSJElSa4w0kRER20XEyX2Pj4mIT27mtWdHxIGjNiipu8waSVUzZyTVwayRxmvUPTK2A06e9VVAZr4kM7854jiSus2skVQ1c0ZSHcwaaYxGnch4K7BPRKyMiLcVy7aJiI9FxI0R8YGICICIuDgilkXE/Ig4NyKui4g1EfHasfwGkiaZWSOpauaMpDqYNdIYjXrVkjcAB2fmYdDbNQpYChwE/Ai4HHgccFlfzWHArpl5cFGz3UwDRMRyYDnAIma+TJikiWXWSKpa5TlTvMaskbrNrJHGaJwn+7wqM2/NzPXASmDPjZ7/LrB3RJwZEU8F7phpZZl5VmYuy8xlZa5TK2nimDWSqjbWnAGzRtImmTXSiMY5kXFv3/0pNtrbIzN/ARwKXAy8Ajh7jGNL6g6zRlLVzBlJdTBrpBGNemjJWmDJMAURsQNwX2aeHxH/BZw74tiSusOskVQ1c0ZSHcwaaYxGmsjIzNsi4vKIuA74DPCpAcp2BVZExPReIKeNMrak7jBrJFXNnJFUB7NGGq9R98ggM0/YaNHFfc+9su/+MX2vOXzU8SR1k1kjqWrmjKQ6mDXS+IzzHBmSJEmSJEmVciJDkiRJkiS1xsiHltQpFm3J/L33Hbn++KP3KDX+La/dtVT9Xmf/V6n6qV12KFWfC0vMV121ptTYUutEjFz61D0eXWro82+9tFT9s3d7bKl6Se1w7MMOK1X/6R9+o1T903d1T3epFSKIhVuMXP7nBz6x1PBvuf7jper/aunvlqqfuv2Xpeo1t7lHhiRJkiRJag0nMiRJkiRJUms4kSFJkiRJklrDiQxJkiRJktQalU9kRMQVVY8hSWaNpDqYNZLqYNZIM6t8IiMzj6x6DEkyayTVwayRVAezRppZHXtk/Kr4uUtEXBoRKyPiuog4uuqxJXWHWSOpDmaNpDqYNdLMFtQ41gnAZzPzjIiYD2w904sjYjmwHGDRwm1raE/ShBg9a2Z+qST1M2sk1cGskTahzomMrwPvjYiFwH9m5sqZXpyZZwFnATxoq12yhv4kTYaRs2bb2N6skTQos0ZSHUbPmnkPMWs0sWq7aklmXgo8Hvgh8L6IeEFdY0vqDrNGUh3MGkl1MGukTattIiMi9gD+JzPfA5wDHF7X2JK6w6yRVAezRlIdzBpp0+o8tOQY4M8iYh3wK8DZRElVOAazRlL1jsGskVS9YzBrpN9Q+URGZm5T/DwPOK/q8SR1k1kjqQ5mjaQ6mDXSzGo7tESSJEmSJKksJzIkSZIkSVJr1HmOjJHlPfcydcO3Gxv/t95yS6n69VuXu4bzs768plT9BX/4OyPXri81stRCMfr8bt6/rtTQz97tsaXq5z9i31L1u577o1L1P/yjh45cO/Xt75YaW2qTmDePeVuV+G4QUWr8p+9a7lyBPznlyFL195f7WsRvfeaXI9fmtdeXG1xqk0xy3X2jl5eoBfiLvR5dqv78Wz9bqv74hx9Tqn79PfeUqle13CNDkiRJkiS1hhMZkiRJkiSpNZzIkCRJkiRJreFEhiRJkiRJao1SExkRsV1EnNz3+JiI+GT5tiTpAWaNpKqZM5LqYNZI41F2j4ztgJNnfZUklWPWSKqaOSOpDmaNNAZlJzLeCuwTESsj4m3Fsm0i4mMRcWNEfCCid42wiHhURFwSEd+IiM9GxC4lx5bUHWaNpKqZM5LqYNZIY7CgZP0bgIMz8zDo7RoFLAUOAn4EXA48LiKuBM4E/ndm/jQingucAbx4cyuOiOXAcoBFlLzguKS2M2skVa2ynCnW90DWxOKqfgdJc199WeP3Gk2wshMZm3JVZt4KEBErgT2B24GDgc8XE4zzgR/PtJLMPAs4C2Db2D4r6FNSu5k1kqo2lpyBDbPmQfN3MGsk9aska/xeo0lWxUTGvX33p4oxArg+M4+oYDxJ3WTWSKqaOSOpDmaNNKSy58hYCywZ4HU3ATtGxBEAEbEwIg4qObak7jBrJFXNnJFUB7NGGoNSExmZeRtweURc13eymk297j7gOcDfR8QqYCVwZJmxJXWHWSOpauaMpDqYNdJ4lD60JDNP2GjRxX3PvbLv/krg8WXHk9RNZo2kqpkzkupg1kjllT20RJIkSZIkqTZOZEiSJEmSpNao4qol2sj6u+4qVX/QlreWqv/Pu+6d/UWSetZPNd3ByKZu/E6p+uMfcl2p+n/mmaXqpa7I9etLfzdo0s7/fEWp+nO+f1mp+uXv+L2Ra9ub8FL3/MFho3/WAeIzW5aqv/tfdhu5dqv/vKrU2Jqde2RIkiRJkqTWcCJDkiRJkiS1hhMZkiRJkiSpNZzIkCRJkiRJrTH0REZEvDoiboiID0TE/4qINwxRu2dEbHzdZEn6DWaNpDqYNZLqYNZI4zXKVUtOBp6Wmd8rHl+48QsiYkFm3r+J2j2BE4D/GGFcSd1i1kiqg1kjqQ5mjTRGQ01kRMS7gL2BCyPivcAvgGWZ+cqIOBf4ObAUuCYiLgT+pShN4PHAW4FHRMRK4LzM/H/j+TUkTRKzRlIdzBpJdTBrpPEbaiIjM/8kIp4KPCEzfxYRJ270kv2AJ2fmVER8AnhFZl4eEdsA9wBvAE7NzGfMNlZELAeWAyxi62HalNRyZo2kOpg1kupg1kjjN+6TfX40M6eK+5cD/xQRrwa228xuUpuVmWdl5rLMXLaQLcfcpqSWM2sk1cGskVQHs0Ya0rgnMu6cvpOZbwVeAmwFfC0iDhjzWJK6y6yRVAezRlIdzBppSKOc7HMgEbFPZq4B1kTEEcABwA+AJVWNKal7zBpJdTBrJNXBrJEGM+49MvqdEhHXRcQq4G7gM8Bq4P6IWBURr61wbEndYdZIqoNZI6kOZo00gKH3yMjMPfvunwucW9w/caPXvWozq3jSsGNK6h6zRlIdzBpJdTBrpPGqco8MSZIkSZKksarsHBkan7/d79Gl6i/6/sdHrj32YYeVGltSjTJLlf/z/zquVP1Jn7po5Nqz9tu71NiS2mP5004qVX/SVZ8ZudaskVokyv2fe7ygXP3xF3125NpPf363UmOvv/PO2V/Uce6RIUmSJEmSWsOJDEmSJEmS1BpOZEiSJEmSpNZwIkOSJEmSJLWGExmSJEmSJKk1ZpzIiIg9I+K6cQwUETdHxA7jWJekyWLWSKqDWSOpDmaNVD33yJAkSZIkSa0xyETGgog4LyJWR8THImJrgIh4UkRcGxFrIuK9EbHlTMunRcRWEXFRRLy0gt9HUnuZNZLqYNZIqoNZI1VokImM/YGzMvMQ4A7g5IhYBJwLPDczHwksAF6+ueV969oG+ATwH5n5npkGjYjlEXF1RFy9jnuH/LUktZBZI6kOZo2kOpg1UoUGmcj4QWZeXtx/P3AUvQ/m9zLzW8Xy84DHz7B82v8HrMjMf59t0Mw8KzOXZeayhWw528sltZ9ZI6kOZo2kOpg1UoUGmcjITTyOzbx2c8unXQ48LSJme52k7jFrJNXBrJFUB7NGqtAgExm7R8QRxf3nAZcBNwJ7RsTDi+V/DFwyw/JpbwRuA95RtnFJE8eskVQHs0ZSHcwaqUKDTGTcALwwIlYD2wPvzMx7gBcBH42INcB64F2bW77R+k4BFkXEP4zrl5A0EcwaSXUwayTVwayRKrRgpicz82bgwM0890Vg6RDL9+x7+KJhmpQ02cwaSXUwayTVwayRqjfIHhmSJEmSJElzwox7ZGhuyPvvL1V/7K6/Mbk7sPkPeXCpsW994QGl6tfuv65U/X4vv6ZUfcyfP3rxOs/HpHrFgnKRPnXjf5WqP+d/Hzty7T3PKJc1t+9b7nff5bK1pern3XRLqXrmjZ4XcUeJnJIaMHX9TaXqzz5kk//RPZD525W7isNNb3xEqfptv1Pu/xB3Pm9VqfrYYovRa39p1qheUz/9aaPjf/KRO4xc+5Hvf6HU2H/wW0eWqo8FC0vV57r7StXXwT0yJEmSJElSaziRIUmSJEmSWsOJDEmSJEmS1BpOZEiSJEmSpNYYaiIjIraLiJP7Hh8TEZ8cch0nRsTDhqmR1C1mjaQ6mDWSqmbOSNUYdo+M7YCTZ33VzE4E/CBKmolZI6kOZo2kqpkzUgWGnch4K7BPRKyMiLcVy7aJiI9FxI0R8YGICICIeGNEfD0irouIs6LnOcAy4APFOrYa4+8iaXKYNZLqYNZIqpo5I1Vg2ImMNwD/lZmHZeafFcuWAqcABwJ7A48rlv9rZv52Zh4MbAU8IzM/BlwNPL9Yx93lfwVJE8iskVQHs0ZS1cwZqQLjONnnVZl5a2auB1YCexbLnxARV0bEGuCJwEHDrDQilkfE1RFx9TruHUObklrOrJFUB7NGUtUqyRkwa9QdC8awjv5PyBSwICIWAe8AlmXmDyLir4FFw6w0M88CzgLYNrbPMfQpqd3MGkl1MGskVa2SnAGzRt0x7B4Za4ElA7xu+kP3s4jYBnjOCOuQ1F1mjaQ6mDWSqmbOSBUYaiIjM28DLi9OQPO2GV53O/AeYA3wn8DX+54+F3iXJ6uRtDlmjaQ6mDWSqmbOSNUY+tCSzDxho0UX9z33yr77pwOnb6L+fOD8YceV1C1mjaQ6mDWSqmbOSOM3jpN9SpIkSZIk1cKJDEmSJEmS1BrjuGqJ5roc/YTFU7f9vNTQu1zxq1L1r3zpF0rVn79+p1L1uX6qRLEnila98v77Gx1/6oYqk5AHAAAgAElEQVRvj1y76MYoNfbac5eWqp/astw51Ha9em2pemL0/1fIqRI5JbXQ+nvuGb24TC2w6Kfl/g/wjiPvLlW/0zvuLFXPnaPXZ5o16pgS/w74g92PKjX0/O22LVV/5spPlKo/eY9y/TNv/ui1A77t7pEhSZIkSZJaw4kMSZIkSZLUGk5kSJIkSZKk1nAiQ5IkSZIktUYjExkRcUUT40rqFrNGUh3MGkl1MGukBzQykZGZRzYxrqRuMWsk1cGskVQHs0Z6QFN7ZJS7JqckDcCskVQHs0ZSHcwa6QELmm5gcyJiObAcYBFbN9yNpEll1kiqg1kjqQ5mjbpizp7sMzPPysxlmblsIVs23Y6kCWXWSKqDWSOpDmaNumLOTmRIkiRJkiRtzIkMSZIkSZLUGk5kSJIkSZKk1mjq8qvbNDGupG4xayTVwayRVAezRnqAe2RIkiRJkqTWcCJDkiRJkiS1xoKmG1AN5s0fvTbXlxp6wS3/U6r+0Yu+V6r+fHYqVU/E6LVZbmipU7LcB2bfF15Tqv6zP1pZqv7Ytx5Wqp6cKlcvqRa7/Z8rStWXzhpKZo2kepT8N9TUL+8oVb/PwoaPQlpf/fca98iQJEmSJEmt4USGJEmSJElqDScyJEmSJElSaziRIUmSJEmSWmPsExkR8dcRcWpx/80R8eRNvOaYiPjkuMeW1A3mjKQ6mDWS6mDWSMOr9KolmfnGKtcvSeaMpDqYNZLqYNZIgxnLHhkR8ZcRcVNEfAHYv2/5uRHxnOL+UyPixoi4DHjWOMaV1B3mjKQ6mDWS6mDWSOWU3iMjIh4F/CGwtFjfNcA3NnrNIuA9wBOB7wAfHmC9y4HlAIvYumybklqsqpwp6swaSYBZI6keZo1U3jj2yDgauCAz78rMO4ALN/GaA4DvZea3MzOB98+20sw8KzOXZeayhWw5hjYltVglOQNmjaQNmDWS6mDWSCWN62SfOabXSNLmmDOS6mDWSKqDWSOVMI6JjEuB4yJiq4hYAjxzE6+5EdgrIvYpHj9vDONK6g5zRlIdzBpJdTBrpJJKnyMjM6+JiA8DK4FbgK9s4jX3FMdrfSoifgZcBhxcdmxJ3WDOSKqDWSOpDmaNVN5YLr+amWcAZ2xi+Yl99y+id6yXJA3NnJFUB7NGUh3MGqmccZ0jQ5IkSZIkqXJOZEiSJEmSpNYYy6ElmuPWTzU29K8etXup+tc/f3mp+juPX1Sq/p7tR5/ru//8r5YaW+qSWFDur6Nb/uLRpeqPfsVjStVv/ei7StXHuvWjF994eamxpdaJGLl03pblLkd535EHlap/+iO3L1V/+wv2L1X/q91Gf+/WnfO1UmNLdSv73SKnRv831PwHbVtq7Fi8uFT9Sd8/qlT9vMXlLpgzb/sHj1wbP1k42BgjjyBJkiRJklQzJzIkSZIkSVJrOJEhSZIkSZJaw4kMSZIkSZLUGk5kSJIkSZKk1nAiQ5IkSZIktUYjExkRsWdE3BAR74mI6yPicxGxVRO9SJpcZo2kqpkzkupg1kgbanKPjH2Bf8vMg4DbgWf3PxkRyyPi6oi4eh33NtKgpIlg1kiq2ow5A2aNpLEwa6RCkxMZ38vMlcX9bwB79j+ZmWdl5rLMXLaQLWtvTtLEMGskVW3GnAGzRtJYmDVSocmJjP4pwilgQVONSJpoZo2kqpkzkupg1kgFT/YpSZIkSZJaw4kMSZIkSZLUGo3sjpSZNwMH9z3+xyb6kDTZzBpJVTNnJNXBrJE25B4ZkiRJkiSpNZzIkCRJkiRJrdGeM93Omz967fqpcmNHlKsvK7O5scu878A2195aqv7OQ3ctVT/v/lLl3L3T6H/269vz6ZIal/eX+7Du9fbrS9VP3fGrUvVbfnnHUvV3vPW3Rq7N7/h/EuqYEt+L1t9zT6mhF166qlT91FS576S/eNpdpeq3umrx6MUNfh2VRhIl/36M0Tf6st8rFixZUqp+1dmPLFW/ze+U+162drfR/yG07vwtBnqd334kSZIkSVJrOJEhSZIkSZJaw4kMSZIkSZLUGk5kSJIkSZKk1nAiQ5IkSZIktUbjExkRcXPTPUiafGaNpDqYNZLqYNao6xqfyJAkSZIkSRrU6Bd4HZ+fbmphRCwHlgMsYutaG5I0kcwaSXUwayTVwaxRpzW+R0Zm/vZmlp+Vmcsyc9lCtqy7LUkTxqyRVAezRlIdzBp1XeMTGZIkSZIkSYNyIkOSJEmSJLWGExmSJEmSJKk1nMiQJEmSJEmt4USGJEmSJElqjblw+dXBrJ9qbuzM5sZuWCwst4msv/2Xpeq3/vq6UvX/fdzDS9Xfs8+9I9fmlt3dbqShRZQr32qrUvXzdt+lVP11ax5Uqv5hi0evzfnl3jupU0pmzbz99ylVf99OJT7swPaf2qJU/d07lCj2a41aJtfd13QLI8u77y5Vf9tjy/0baqd/+EWp+nu323Hk2lg/2OvcI0OSJEmSJLWGExmSJEmSJKk1nMiQJEmSJEmt4USGJEmSJElqjVonMiJiu4g4uc4xJXWPWSOpDmaNpDqYNdJvqnuPjO0AP4SSqmbWSKqDWSOpDmaNtJG6JzLeCuwTESsj4m01jy2pO8waSXUwayTVwayRNrKg5vHeABycmYfVPK6kbjFrJNXBrJFUB7NG2kjdExkDi4jlwHKARWzdcDeSJpVZI6kOZo2kOpg16oo5e9WSzDwrM5dl5rKFbNl0O5ImlFkjqQ5mjaQ6mDXqironMtYCS2oeU1L3mDWS6mDWSKqDWSNtpNaJjMy8Dbg8Iq7zRDWSqmLWSKqDWSOpDmaN9JtqP0dGZp5Q95iSuseskVQHs0ZSHcwaaUNz9hwZkiRJkiRJG3MiQ5IkSZIktcacvfyqxihi9NL580sNve639y9Vf/NLs1T9fq/4Zqn6Hb++28i1P/+f9aXGljoly33W7//vn5aq/+TVnypV/4xdH1Wqvox5eWdjY0utUzJr+O+flSo/81MrStWfsueRpeofVKL2FrNGqk/Jf4N9+slvL1V/yktKZs13bxm5dv79g2WNe2RIkiRJkqTWcCJDkiRJkiS1hhMZkiRJkiSpNZzIkCRJkiRJrVHbREZE/HVEnFrXeJK6yayRVAezRlIdzBpp09wjQ5IkSZIktUalExkR8ZcRcVNEfAHYv1h2WER8LSJWR8QFEfHgKnuQNPnMGkl1MGsk1cGskWZX2URGRDwK+ENgKfAs4LeLp/4d+PPMPARYA7xpM/XLI+LqiLh6HfdW1aakljNrJNXBrJFUB7NGGkyVe2QcDVyQmXdl5h3AhcBiYLvMvKR4zXnA4zdVnJlnZeayzFy2kC0rbFNSy5k1kupg1kiqg1kjDaDqc2RkxeuXJDBrJNXDrJFUB7NGmkWVExmXAsdFxFYRsQR4JnAn8IuIOLp4zR8Dl2xuBZI0ALNGUh3MGkl1MGukASyoasWZeU1EfBhYCdwCfKV46oXAuyJia+C7wIuq6kHS5DNrJNXBrJFUB7NGGkxlExkAmXkGcMYmnnpsleNK6hazRlIdzBpJdTBrpNlVfY4MSZIkSZKksXEiQ5IkSZIktUalh5ZojsjRT3y8/q67Sg09/+JrStXvc3Gpcm4+/chS9fcecPfotadHqbElDSHXlyp/xq6PKlX/vBt/VKr+w8990si1cdPlpcaWNLipn91Wqv6UPct9L/nhxw8qVT//kgeNXLvuP75WamxJg1v/89tL1b92/yeUqv/WioNL1S++afRL/65bcdlAr3OPDEmSJEmS1BpOZEiSJEmSpNZwIkOSJEmSJLWGExmSJEmSJKk1nMiQJEmSJEmt4USGJEmSJElqDScyJEmSJElSayxouoHNiYjlwHKARWzdcDeSJpVZI6kOZo2kOpg16oo5u0dGZp6Vmcsyc9lCtmy6HUkTyqyRVAezRlIdzBp1xZydyJAkSZIkSdqYExmSJEmSJKk1Gp/IiIizI2JZ031ImmxmjaQ6mDWSqmbOSHPgZJ+Z+ZKme5A0+cwaSXUwayRVzZyR5sAeGZIkSZIkSYNyIkOSJEmSJLVGZGbTPcwqIn4K3DLDS3YAflZiiC7Xt7n3putnq90jM3cccd1qgFkzZ8due33VY5s1LTPhWdPm3puun+u9mzUtM8ezZq5v73O5vs29D1I/UNa0YiJjNhFxdWaOfMKbLte3ufem68uOrfZp8/Zatr7NvTdd33Tvap+mt5k2b+9trm9z72ont/d21re593HUT/PQEkmSJEmS1BpOZEiSJEmSpNaYlImMs6xv5dhtry87ttqnzdtr2fo29950fdO9q32a3mbavL23ub7Nvaud3N7bWd/m3sdRD0zIOTK6LCJ+lZnb9D0+EViWma8cw7ovBk7NzKs3Wv5K4BRgH2DHzCxzshdJc1xDOfMBYBmwDrgKeFlmris7nqS5q6GsOYde1gTwLeDEzPxV2fEkzV1NZE3f82cCL+ofX6OZlD0yVK/LgScz81mQJamMDwAHAI8EtgJe0mw7kibUazPz0Mw8BPg+UPofMpK0KRGxDNiu6T4mhRMZEywidoyI8yPi68XtccXyR0fEFRFxbfFz/2L5VhHxoYhYHREfpvePh9+Qmddm5s31/SaS5qoKc+bTWaC3R8Zutf1SkuacCrPmjuL1UbzGXZWlDqsqayJiPvA24PW1/TITbkHTDai0rSJiZd/j7YELi/v/Avy/zLwsInYHPgs8ArgReHxm3h8RTwb+Dng28HLgrsw8JCIOAa6p7beQNJc1ljMRsRD4Y+A1Y/2NJM1FjWRNRKwAng58E/jTcf9SkuacJrLmlcCFmfnj3rypynIio/3uzszDph9MH+NVPHwycGDfh2XbiFgCPAg4LyL2pfc/DwuL5x8PvB0gM1dHxOrq25fUAk3mzDuASzPzK+P4RSTNaY1kTWa+qPjf0jOB5wIrxvYbSZqLas2aiHgYcDxwzNh/kw5zImOyzQOOyMy7+xcWJ5n5cmYeFxF7Ahf3Pe0ulZKGUVnORMSbgB2Bl42lU0ltVul3msycKnYL/zOcyJC6rIqsWQo8HPhOMUGydUR8JzMfPq6mu8hzZEy2z9F30qqImJ55fBDww+L+iX2vvxR4fvHag4FDqm9RUstVkjMR8RLgWOB5mbl+vC1LaqGxZ030PHz6PvBMeruPS+qusWdNZn4qM3fOzD0zc096h6I4iVGSExmT7dXAsuLkM98E/qRY/g/A/4mIy4H5fa9/J7BNsUvU6+mdYO83RMSrI+JWeiffWx0RZ1f2G0ia6yrJGeBdwEOBr0bEyoh4YzXtS2qJKrIm6O0qvgZYA+wCvLmqX0BSK1T1vUZjFr0TwkuSJEmSJM197pEhSZIkSZJaw4kMSZIkSZLUGk5kSJIkSZKk1nAiQ5IkSZIktYYTGZIkSZIkqTWcyJAkSZIkSa3hRIYkSZIkSWoNJzIkSZIkSVJrOJEhSZIkSZJaw4kMSZIkSZLUGk5kSJIkSZKk1nAiQ5IkSZIktYYTGZIkSZIkqTWcyJAkSZIkSa3hRIYkSZIkSWoNJzIkSZIkSVJrLGi6AY0mIg4Fji4efiUzVzXZj6TJY85IqoNZI6kOZs1kcY+MFoqI1wAfAHYqbu+PiFc125WkSWLOSKqDWSOpDmbN5InMbLoHDSkiVgNHZOadxePFwFcz85BmO5M0KcwZSXUwayTVwayZPO6R0U4BTPU9niqWSdK4mDOS6mDWSKqDWTNhWn2OjIgI4ALgtMy8oel+arQCuDIiLige/z5wToP9SBOto1ljzkg1M2sAs0aqnFkDmDWt1+pDSyLiWHob4Icz80+b7qdOEXE4cBS9mcRLM/PahluSJlZXs8ackepl1pg1Uh3MGrNmErR9IuMjwHuBtwMHZub9DbdUuYiYB6zOzIOb7kXqiq5ljTkjNcOskVQHs0aToLXnyIiIHYCDMvMi4AvAcQ23VIvMXA+siojdm+5F6oIuZo05I9XPrJFUB7NGk6K1ExnAC4APFvdXACc12EvddgGuj4gvRsSF07emm+qSiDguIrZpug/VoqtZY87MAWZNp5g1Zk1jzJpOMWvMmsaMM2tae2hJRKwBnpqZPywerwKekZk/aLaz6kXE72xqeWZeUncvXRQR+wA3Aq/KzHc13Y+q1dWsMWeaZ9Z0i1mzIbOmPmZNt5g1GzJr6jPurGnlREZEbAc8NzPf3bfsd4GfedIWVS0izgASeEpmPrrpflQds0ZNMmu6w6xRk8ya7jBr1KRxZ00rDy3JzNuB6zZa9nlg62Y6qkdEXFb8XBsRd/Td1kbEHU331wURMR84Hvh74JcRcWjDLalCXcwac2ZuMGu6xawxa5pi1nSLWWPWNKWKrGnlREbhzAGXTYzMPKr4uSQzt+27LcnMbZvuryOeDlyRmWvpne35JQ33o+p1KmvMmTnDrOkes8asaYJZ0z1mjVnThLFnzYLSLdUsIo4AjgR2jIjX9T21LTC/ma7qFxFHAftm5ori7MNLMvN7TffVAScB/7e4fwHwloj408y8r8GeVAGzxpxpmFnTEWaNWdMws6YjzBqzpmFjz5o27pGxBbANvUmYJX23O4DnNNhXbSLiTcCfA6cVi7YA3t9cR91QHFe4XWZ+BSAz7wE+Bjyx0cZUlU5njTnTHLOmc8was6YRZk3nmDVmTSOqypq2nuxzPvDhzJz4D92mRMRKYClwTWYuLZatzsxDmu1MmixdzhpzRqqPWWPWSHUwa8yaSdK6Q0sAMnMqIrZvuo8G3ZeZGREJEBGLm25o0kXE4TM9n5nX1NWL6tPxrDFnGmDWdJNZY9bUzazpJrPGrKlblVnTyomMwrURcSHwUeDO6YWZ+fHmWqrNRyLi3cB2EfFS4MXAexruadJNH9O1CFgGrAICOAS4Ejiqob5Uva5mjTnTDLOmu8was6ZOZk13mTVmTZ0qy5pWHloCEBErNrE4M/PFtTfTgOKaz0+htyF8trh0kioWER8CzsjMNcXjg4FTM/PERhtTZbqcNeZMc8ya7jFrzJommDXdY9aYNU2oImtaO5EhiIht6durJjN/3mA7nRARKzPzsNmWSZPCnGmGWaOuMWuaYdaoa8yaZlSRNa09tCQiFtG7jMtB9HZVAaAjs4kvA94M3A2spzermMDeA9ZvAexXPLwpM9dV0eeEuiEizqZ3luME/gi4odmWVKWuZk3ZnCnWYdaMzqzpGLPGrGmIWdMxZo1Z05CxZ00bL7867X3AzsCxwCXAbsDaQQoj4qERcU5EfKZ4fGBEnFRZp+N3KnBQZu6ZmXtn5l6ZOegkxjHAt4F/A94BfCsiHl9dqxPnRcD1wGuAU4BvFss0ubqaNSPnDJg1Y2DWdI9ZY9Y0wazpHrPGrGnC2LOmtYeWRMS1mbl0+rI5EbGQ3rFOs16PtvjwrQD+MjMPjYgFwLWZ+ciq+x6HiLgIeFZm3jVC7TeAEzLzpuLxfsAHM/NRY25TmghdzZoyOVPUmzXSEMwas0aqg1lj1kyK1h5aAkzvynN7cbKQnwB7Dli7Q2Z+JCJOA8jM+yNiqoIeq3IacEVEXAncO70wM189QO3C6Q9gUfOtIsAGFhF7APtm5hciYitgQWYONJPbdhHxOOCvgT3Y8Pi6gWd01TpdzZoyOQNmTSlmTSeZNWZN7cyaTjJrzJraVZE1bZ7IOCsiHgycDlwIbAP81YC1d0bEQ+gdn0NEPBb4ZSVdVuPdwJeANfSO8RrG1RFxDr3dygCeD3xj0OLoXa5oObA9sA+93dHeBTxpyD7a6hzgtfTes7YEt8rpataUyRkwa8oya7rHrDFrmmDWdI9ZY9Y0YexZ0+ZDS/bKzO/NtmwztYcDZwIHA9cBOwLHZ+aqSpods4i4IjOPHLF2S+AV9K7ZG8ClwDsy894ZCx+oXwk8GrgyM5cWy9a0YZeycYiIKzPzMU33ofp0NWvK5ExRb9aUYNZ0j1kzcr1ZU4JZ0z1mzcj1Zk0JVWRNmycyrsnMwzda9o1BjlMqNsQpYH96G+JNwLxBN8SmRcQZwC3AJ9hw16gZLx8UEfOB8zLzj0qMfWVmPqbv+LoFwDWZecio62yTiHgrMB/4OBu+99c01pQq1dWsGTVnilqzpiSzpnvMGrOmCWZN95g1Zk0Tqsia1h1aEhEH0Ltc0IMi4ll9T21L3yWEZvHV4gN8fd96rwEO33zJnHJC8fO0vmWzXj4oM6ciYseI2CIz7xtx7Esi4i+ArSLid4GT6QVCV0zPJC7rW5bArCdIUruYNaPlDJg1Y2LWdIRZY9Y0zKzpCLPGrGnY2LOmdRMZ9GYAnwFsBzyzb/la4KUzFUbEzsCu9DagpfRmEqH3Ad56/K1WIzP3KlF+M3B5RFwI3Nm3zn8asP4N9K49vQZ4GfBp4OwS/bRKZj6h6R5Um05nTcmcAbOmFLOmU8yacm7GrBmZWdMpZk05N2PWjKyKrGnzoSVHZOZXh6x5IXAivZmgr/PAh3AtcG5mfnysTY5ZRDwxM7+00Szqrw3Sf0S8aTO1f1O2vy6IiIcCfwc8LDOfFhEHAkdk5jkNt6aKdC1rxpEzxXrMmhLMmu4xazZk1tTDrOkes2ZDZk09qsiaNk9k/APwFuBu4CLgUOCUzHz/ALXPzszzK25x7CLibzLzTRGxYhNPZ2a+eIB1LM3Ma0v08D2KMxVvNPhAl84pet9U/ay9zwXR4utnazRdy5px5EyxHrOmBLOme8yaDZg1NTFruses2UArsqbtOQPVZE0bDy2Z9pTMfH1EHAfcChwPfBmY9UMI7BYR29KbRXwPveO63pCZn6us2zEoPoTzgM9k5kdGXM0/RcQuwEeBD2Xm9bMVbKT/uKZF9N737Yeo/+RG9ccBPxq0OCKOBq7IzKm+ZYfXeFKqNl8/W6PpVNaMKWfArCnLrOkes2Y0Zk05Zk33mDWjaTJrSuUMTGjWZGYrb8D1xc/3AE8t7q8asHZV8fNYetdPPpTeWWMb/70G7P/SkvU7A68GLqd3nNbpJdd3WYnaecCXhnj9XcAlwEP7ltX2ZwdcDDxkekzgscAlTW8T3ir9M+9k1pTNmWIdZs3o/Zo1HbuZNaXWYdaM3q9Z07GbWVNqHXMia4bNmaJm4rKmzXtkfCIibqS3W9TJEbEjcM+AtdPHdT0dWJGZqyIiZiqYYz4fEacCH2bDk83Mevmg4nU/Ad4eEV8GXg+8kd4uZrOK3vWjp82jN7u4ZMC+N2VfYPchXn8T8Dbg4og4KTOv4IE/zzq8jl5w7xMRl9O7fvZzahxf9etq1pTKmeK1Zs3ozJruMWvMGrNGdTBr2p81w+YMTGDWtPYcGQAR8WDgjuxdEmcxsKTYwGarW0HvzLt70ZtJnA9cnANcP3kuKI6x2ljmYMdYPQJ4Lr0N5zbgQ8D5mfk/A4795b6H99M7g+8/ZuZNA9avpXeMVxQ/fwKclgMebxfFta8jYl96QfRe4MW50fWwq1Qc0/Xr62dn5rq6xlYzupg1ZXKmqDdrSjJruses+TWzxqxRhcyaX2tF1pTNmWIdE5c1rZzIiIitgX0zc1Xfst2Bqcz84QD184DDgO9m5u0R8RBg18xcXVnTc0REfA34IPDRzBzq2Kq5ICKuzcylxf3F9E4a86zMrHzvorLbndrHrBmdWVNqbLOmY8ya0Zk1pcY2azrGrBmdWVNq7Eqypq0TGQuBG4FDMvPOYtnngL/IzKsHqA/g+cDemfnm4o3cOTOvGnD8UvXFOg4Fji4efqX/D3aA2kXAycBR9GblvgK8KzMH3S1sZBHxupmez1mupdz33u2VmX87ynu3iXXunpnfH7V+iHFKbXdqny5nTZM5U4xv1pg1ndH2rGnrd5pifLPGrOmMJrOmy/9+KsYfOWuqyJliva3Omnlj6q9WxW4oF9DbvWd6RmfHId6IdwBHAM8rHq8F/m2mgoh4XETMH7V+o3W9BvgAsFNxe39EvGrQeuDfgYOAM4F/BQ4E3jfLmB8pfq6JiNV9tzURMcws6jLg5fR2K9sV+JNi/CUMdpzX9Ht3QvF4oPcuIl5f/DwzIt7efwNOHaL/kY1hu1PLdDxrhs6ZYkyzpiSzpnvanDVNfKcpxjVrSjJruqfurJlD32mg3VkzUs4UfU9u1mRNZyod9w04gN5MHMDpwKuHqJ0+W+q1fctmPFsvcCRw1qj1G61rNbC47/FiYPUQ9b8x1gD971L83GNTtyHG/hy94+imHy8BLqryvS9ec1vx8xTghRvf2rDdeWvnratZM0rOFK8xaxre7ry189bWrCmTM5sbx6wxa7zNzT/zYbf3ceVM8drOZk3JjJ7YrGntVUsy88aIICL2ozezd9QQ5euK2cEEiN7ZetfPMt4VEXHXqPUbCaD/urlTxbJBXRsRj83MrxXjP4beZYA2KzN/XPy8ZYhxNmV34L6+x/cBew5RP+p7998RsQfwIuAJQ4w3ViW3O7VQh7Nm6JwBs2ZczJruaXHW1P6dBsyacTFruqfOrJlD32mg3VlT5r2b2Kxp7URG4RzgbHqzcb8You7t9HZv2SkizqB39tnTZyvKzJVl6vusAK6MiAuKx79P73cZ1GOAF0TE9DFNuwM3RMSaXpt5yMYF8cDZbn/jqaJm2wHHfh9wVdF7AscB5w3R+6jv3TuBi4C9gf7dkKbP3jvQGYc3JyJ2zgHO1lwYdbtTe3Uxa4bOGTBrZmPWaBZtzJrav9OAWTOTIXMGzJouqi1r5sh3Gmh31pR57yY2a1p5ss9p0TsD6o+BZ2fmF4asPQB4Er0/xC9m5g011x9ObyYqgEsz89ohaveY6fkxzBjONv7hPHCinaF6L+pHfu8i4p2Z+fJhxhtwvZ/KzN8b8LUjb3dqpy5mTdM5U/Rg1pg1ndLWrGnzd5qih4nKmmFypni9WdMxTWVNl//9VPQwctaM4b2buKxp9USGJEmSJEnqllZetUSSJEmSJHXTRExkRMRy69s3dtvry46t9mnz9lq2vs29N2V0n7sAAB9HSURBVF3fdO9qn6a3mTZv722ub3Pvaie393bWt7n3cdRPm4iJDKDsm9Hl+jb33nS9f+F3T5u317L1be696fqme1f7NL3NtHl7b3N9m3tXO7m9t7O+zb2Pox6YnIkMSZIkSZLUAa042ecWsWUuYvFmn1/HvSxky80+v98hd232OYCf3jbFjg+Zv9nnv7V66xnrZxt/Nk3Wt7n3putnq72HO7kv7x3m+tZqWNmsWXzgzHl61y/u/f/bu/dgy86yTsC/ty/pTtNNAgIDGCQSEISIgTSRcA2XGWQGHINxuFiMsQp7MCIFCIpoMZY1SEa8lFeoBiFRgiIgVy+gyDVyC9AkUUAugSEgBGKQJEAnffqbP/pEDk1fztnf3mv3Ovt5qqjss/Z61/vtqrN/7PP22mtl2y0OX3/d5w7fO0luvPH6bN58+H3q2iNnnaw4NrOit17WjE9v1pxyz+uOePyvXL0/t/quw/9b1acvP/LdAm9o38xxtfWwz7f9+w/73Jjfq/OuP9bXLmvGpzdrtt/98O/1JPn6NTdk2y2OO+zz131u+2Gfu+GG63PccUf+3JMjfK451t8vs6wf89pXU7/arNk08QoGtDU3yw/Vwyauf/Ob9xx9pyN4xEmnd9Vn/1JfPaP0vvbWeS+BNerNmvv8Wd97/T3POKOrftM/fLCrnnGSNePTmzV/8Tfv6er/uLs+vKt+//XXd9UzTrJmfHqz5n5/fkNX/3c97b5d9Rvf9qGuesZptVnjqyUAAADAaBhkAAAAAKNhkAEAAACMxkSDjKo6sarOW/HzWVX1psPs+5KquvukCwQWl6wBZk3OAEOQNTBdk56RcWKS8466V5LW2pNaa/88YR9gsckaYNbkDDAEWQNTNOkg4/wkp1TVnqp6wfK27VX16qr6WFVdVFWVJFX19qraWVUbq+qCqrq8qi6rqqdP5RUA65msAWZNzgBDkDUwRZPefvXZSU5trZ2WHDg1Ksm9ktwjyReSXJzk/knevaLmtCTf3Vo7dbnmxCM1qKpdSXYlydZsm3CZwMjJGmDWZp4zy/vIGlhssgamaJoX+3x/a+3K1tr+JHuSnHzQ859Ocqeq+v2q+uEkXzvSwVpru1trO1trOzdnyxSXCYycrAFmbao5k8ga4JBkDUxomoOMvSseL+Wgsz1aa9ck+cEkb0/ys0leMsXewOKQNcCsyRlgCLIGJjTpV0uuTbJjLQVVdaskN7TWXlNVn0pywYS9gcUha4BZkzPAEGQNTNFEg4zW2tVVdXFVXZ7kb5L81SrKvjvJy6rqprNAfmmS3sDikDXArMkZYAiyBqZr0jMy0lp7wkGb3r7iuaeseHzWin3uPWk/YDHJGmDW5AwwBFkD0zPNa2QAAAAAzJRBBgAAADAaE3+1ZFCV1KbJl/rf7v2Irva//qk3dtU/504/1FWf1vrqgUF84LSNXfVv/cIfd9U/4vanddUD4/C4Oz+kq/41n3xbV/3ZJ53RVQ8MpPNvqL/+zTO72m97zhe66jf2RRXrnDMyAAAAgNEwyAAAAABGwyADAAAAGA2DDAAAAGA0Zj7IqKp/nHUPAFkDDEHWAEOQNXBkMx9ktNbuN+seALIGGIKsAYYga+DIhjgj47rl/96uqt5ZVXuq6vKqeuCsewOLQ9YAQ5A1wBBkDRzZ5DcWXrsnJHlza+15VbUxybYj7VxVu5LsSpKtR94VYCVZAwxB1gBDkDVwCEMOMj6Q5KVVtTnJ61pre460c2ttd5LdSXLzDbdsA6wPWB8mz5qSNcCqyRpgCP6GgkMY7K4lrbV3JnlQks8n+dOq+p9D9QYWh6wBhiBrgCHIGji0wQYZVXXHJFe11l6c5I+T3Huo3sDikDXAEGQNMARZA4c25FdLzkryrKq6Mcl1SUwTgVk4K7IGmL2zImuA2Tsrsga+w8wHGa217cv/vTDJhbPuBywmWQMMQdYAQ5A1cGSDfbUEAAAAoJdBBgAAADAaQ14jY3Itafv2TVy+70tXdbV/zvee0VW/8RYndtV/6kV36Kr/T6/YOnHtjku/1NV73xWf7aqHwVVNXnrccV2tH3H707rqa8uWrvrnfPR9XfXnn/6QiWuXrrmmqzeMSW3YkA3bd0xev6nv49tjTnlgV/3Pf/JDXfW/ded7dNUDq9T5N9SJf/qevv4vn/wzVZJkw8au8hdd8Y6u+vNOfeTEtfuvvbarN0fnjAwAAABgNAwyAAAAgNEwyAAAAABGo2uQUVUnVtV5K34+q6re1L8sgG+RNcCsyRlgCLIGpqP3jIwTk5x31L0A+sgaYNbkDDAEWQNT0DvIOD/JKVW1p6pesLxte1W9uqo+VlUXVR24BUBVnV5V76iqD1bVm6vqdp29gcUha4BZkzPAEGQNTEHvIOPZST7VWjuttfas5W33SvK0JHdPcqck96+qzUl+P8k5rbXTk7w0yfM6ewOLQ9YAsyZngCHIGpiCvhuRH9r7W2tXJklV7UlycpKvJjk1yd8tDxg3JvnXIx2kqnYl2ZUkW7NtBssERk7WALM2lZxZrv9W1tTNZrRcYKRmkzU+17COzWKQsXfF46XlHpXkn1prZ672IK213Ul2J8nN65ZtqisE1gNZA8zaVHIm+fasOWHjrWQNsNJMssbnGtaz3q+WXJtkxyr2+3iSW1fVmUlSVZur6h6dvYHFIWuAWZMzwBBkDUxB1yCjtXZ1kour6vIVF6s51H43JDknyf+tqo8k2ZPkfj29gcUha4BZkzPAEGQNTEf3V0taa084aNPbVzz3lBWP9yR5UG8/YDHJGmDW5AwwBFkD/Xq/WgIAAAAwGIMMAAAAYDQMMgAAAIDRmMXtV489bb53Hlq65pqu+hff521d9ef/8tkT1+674rNdvWF0OvKi7d179J1mqLf/L3zsnK76G3/81hPX3mr3e7p6w5i0/fuz/9pr572Mif3WnftunFCn99U/4aI3T1x70d1O6uoNrEHv32Btqav8xlRXfW07fvLiEWf8WDgjAwAAABgNgwwAAABgNAwyAAAAgNFY8yCjqp5aVR+tqouq6keq6tlrqD25qg6+bzLAd5A1wBBkDTAEWQPTNcnFPs9L8sjW2hXLP7/h4B2qalNrbd8hak9O8oQkr5igL7BYZA0wBFkDDEHWwBStaZBRVS9Kcqckb6iqlya5JsnO1tpTquqCJP+W5F5JPlRVb0jyu8ulLcmDkpyf5Purak+SC1trvzOdlwGsJ7IGGIKsAYYga2D61jTIaK09uap+OMlDWmtfqapzD9rl+5I8vLW2VFVvTPKzrbWLq2p7km8meXaSZ7bWHnW0XlW1K8muJNmabWtZJjBysgYYgqwBhiBrYPqmfbHPV7X2Hzf8vTjJb1fVU5OceJjTpA6rtba7tbaztbZzc7ZMeZnAyMkaYAiyBhiCrIE1mvYg4/qbHrTWzk/ypCTHJ3lvVd1tyr2AxSVrgCHIGmAIsgbWaJKLfa5KVZ3SWrssyWVVdWaSuyX5XJIds+oJLB5ZAwxB1gBDkDWwOtM+I2Olp1XV5VX1kSTfSPI3SS5Nsq+qPlJVT59hb2BxyBpgCLIGGIKsgVVY8xkZrbWTVzy+IMkFy4/PPWi/nzvMIR621p7A4pE1wBBkDTAEWQPTNcszMgAAAACmyiADAAAAGI2ZXeyTFTZs7Co//8z/0lV/u9d/eeLaz/38aV296+I9XfXAcE48+3Nd9X/5qVdMXHv27jO6egMjcuknusrP2f6FiWsvqjt09U5rffXA6nX+DfW00x7VVf+CD79+4tpn3OkBXb2zf+no+yw4Z2QAAAAAo2GQAQAAAIyGQQYAAAAwGgYZAAAAwGgccZBRVSdX1eXTaFRVn6mqW03jWMD6ImuAIcgaYAiyBmbPGRkAAADAaKxmkLGpqi6sqkur6tVVtS1JquphVfXhqrqsql5aVVuOtP0mVXV8Vf1tVf30DF4PMF6yBhiCrAGGIGtghlYzyLhrkt2ttXsm+VqS86pqa5ILkjy2tfYDSTYl+ZnDbV9xrO1J3pjkFa21Fx+paVXtqqpLquqSG7N3jS8LGCFZAwxB1gBDkDUwQ6sZZHyutXbx8uOXJ3lADrwxr2it/cvy9guTPOgI22/y+iQva639ydGattZ2t9Z2ttZ2bs6Wo+0OjJ+sAYYga4AhyBqYodUMMtohfq7D7Hu47Te5OMkjq+po+wGLR9YAQ5A1wBBkDczQagYZ31NVZy4/fnySdyf5WJKTq+rOy9ufmOQdR9h+k+cmuTrJH/UuHFh3ZA0wBFkDDEHWwAytZpDx0SQ/WVWXJrllkhe21r6Z5KeSvKqqLkuyP8mLDrf9oOM9LcnWqvqNab0IYF2QNcAQZA0wBFkDM7TpSE+21j6T5O6Hee6tSe61hu0nr/jxp9aySGB9kzXAEGQNMARZA7O3mjMyAAAAAI4JBhkAAADAaBzxqyVMyf6lrvKlL13VVX/lfSevfdWVL+zq/djvfXBXfdt3Y1d92sEXjF6jnotDd7aGobW9ffebP/ukM6a0krX7xo/29T7l2R/tqv/Cfa/tqocxqc3HddW3G2/oqj/noY+fuPZ+e/re6+976G276peu/reu+mzY2Fffo+/jLKxd299VvvTVr3bV/0JH1nzpKbfv6n3Cp/d11R/1PjhHcfxbPtJ3gB57V7d4Z2QAAAAAo2GQAQAAAIyGQQYAAAAwGgYZAAAAwGisaZBRVSdW1Xkrfj6rqt60xmOcW1V9Vz8B1jVZAwxB1gCzJmdgNtZ6RsaJSc476l5Hdm4Sb0TgSGQNMARZA8yanIEZWOsg4/wkp1TVnqp6wfK27VX16qr6WFVdVHXgfpVV9dyq+kBVXV5Vu+uAc5LsTHLR8jGOn+JrAdYPWQMMQdYAsyZnYAbWOsh4dpJPtdZOa609a3nbvZI8Lcndk9wpyf2Xt/9Ba+0+rbVTkxyf5FGttVcnuSTJTywf4xuHa1RVu6rqkqq65MbsXeMygZGTNcAQZA0wa4PlTCJrWBzTuNjn+1trV7bW9ifZk+Tk5e0Pqar3VdVlSR6a5B5rOWhrbXdrbWdrbefmbJnCMoGRkzXAEGQNMGszyZlE1rA4Nk3hGCtHfUtJNlXV1iR/lGRna+1zVfWrSbZOoRewuGQNMARZA8yanIFOaz0j49okO1ax301vuq9U1fYk50xwDGBxyRpgCLIGmDU5AzOwpkFGa+3qJBcvX4DmBUfY76tJXpzksiSvS/KBFU9fkORFLlYDHI6sAYYga4BZkzMwG2v+aklr7QkHbXr7iueesuLxryT5lUPUvybJa9baF1gssgYYgqwBZk3OwPRN42KfAAAAAIMwyAAAAABGYxp3LWEd+x8nndlVf+1j791Vv+VJ/9pX/8jPd9XXpsnfIvXN6uoNrN6OD1zZVf/02/5dV/2zct+uehiTduMNXfW1pfOWkJsn///m9z/qTl2tv7Hztl31n330Xbrqv+9pH+yqT3X8G+Z+n2sYWGtd5T2f45Nk/xe+OHHtbf+o73PJhrue0lV/5ws/3VX/ibdu7qpvS0td9avhjAwAAABgNAwyAAAAgNEwyAAAAABGwyADAAAAGI25DDKq6h/n0RdYLLIGGIKsAYYga+Bb5jLIaK3dbx59gcUia4AhyBpgCLIGvmVeZ2RcN4++wGKRNcAQZA0wBFkD39J3c90ZqqpdSXYlydZsm/NqgPVK1gBDkDXAEGQNi+KYvdhna213a21na23n5myZ93KAdUrWAEOQNcAQZA2L4pgdZAAAAAAczCADAAAAGA2DDAAAAGA05nX71e3z6AssFlkDDEHWAEOQNfAtzsgAAAAARsMgAwAAABiNTfNeAOvbzf/yQ131b/zt93TVP2rf6V31bd++yWtb6+oNrN7+r13bVX/7jUtTWglwNG3v3q76pX/6+JRWsnbHXfn5rvpPv+zDXfWPeMppXfVdfK5hZHo+x0+jvkdvzv3e7fd01T/i68d+1jgjAwAAABgNgwwAAABgNAwyAAAAgNEwyAAAAABGY+qDjKr61ap65vLjX6uqhx9in7Oq6k3T7g0sBjkDDEHWAEOQNbB2M71rSWvtubM8PoCcAYYga4AhyBpYnamckVFVv1xVH6+qv09y1xXbL6iqc5Yf/3BVfayq3p3kMdPoCywOOQMMQdYAQ5A10Kf7jIyqOj3J45Lca/l4H0rywYP22ZrkxUkemuSTSV65iuPuSrIrSbZmW+8ygRGbVc4s18kaIImsAYYha6DfNM7IeGCS17bWvt5a+1qSNxxin7sluaK19onWWkvy8qMdtLW2u7W2s7W2c3O2TGGZwIjNJGcSWQN8G1kDDEHWQKdpXeyzTWkfgMORM8AQZA0wBFkDHaYxyHhnkrOr6viq2pHk0YfY52NJvreqTln++fFT6AssDjkDDEHWAEOQNdCp+xoZrbUPVdUrk+xJ8tkk7zrEPt9c/r7WX1XVV5K8O8mpvb2BxSBngCHIGmAIsgb6TeX2q6215yV53iG2n7vi8d/mwHe9ANZMzgBDkDXAEGQN9JnWNTIAAAAAZs4gAwAAABiNqXy1hPWrNvX9inzxZ3Z21T/6R+7SVb//gcd11V/7PZPftmrpr9/b1RtYvesf+v1d9U+861Jf/3N+oKv+qntP/u8KN/yBrIHV2rBjR1f9jaf3fS75gff1ZdVJOz7XVb/h1t81cW1d2feZCoZWW/puP9v27u1oXl29N3Su/QE/97+66r/+5L7zHdqGyV//vr9Y3ecaZ2QAAAAAo2GQAQAAAIyGQQYAAAAwGgYZAAAAwGgYZAAAAACjMZdBRlWdXFUfraoXV9U/VdVbqur4eawFWL9kDTBrcgYYgqyBbzfPMzLukuQPW2v3SPLVJD82x7UA65esAWZNzgBDkDWwbNMce1/RWtuz/PiDSU5e+WRV7UqyK0m2ZtuwKwPWE1kDzNoRcyaRNcBUyBpYNs8zMvaueLyUg4YqrbXdrbWdrbWdm7Nl2JUB64msAWbtiDmTyBpgKmQNLHOxTwAAAGA0DDIAAACA0ZjLNTJaa59JcuqKn39zHusA1jdZA8yanAGGIGvg2zkjAwAAABgNgwwAAABgNAwyAAAAgNGYyzUyGI+2b19X/W1feElXfW3sm7V98g9PPfpOR3Czf5m8//7julrDYqnqKt9x2VVd9W1L3y3qtl59Y1f99lOvn7h2w/F9OQ2LZP/1X++qP+6q67rqr/vKCV31dbvbdNV/43tOnLh2/1X+bGBklpbmvYKJtaX9XfUbb2hd9Utb+z6Xbej7WLS6HrNvAQAAADAdBhkAAADAaBhkAAAAAKNhkAEAAACMxtwHGVX1mXmvAVj/ZA0wBFkDDEHWsOjmPsgAAAAAWK1j4T5KXz7UxqralWRXkmzNtkEXBKxLsgYYgqwBhiBrWGhzPyOjtXafw2zf3Vrb2VrbuTlbhl4WsM7IGmAIsgYYgqxh0c19kAEAAACwWgYZAAAAwGgYZAAAAACjYZABAAAAjIZBBgAAADAaBhkAAADAaGya9wJY3zacsKOrvm7eV7/5qs1d9Rv2dRS3rtawWFrfG2bfbW7eVb9x08au+v2b+/5d4KtX3GLi2qW9/q8cBrO0v6++97PB5r73e3UuH8ak7ev5IN/bvO/N3paWuuq3X/bFrvqvnHpSV/3ekyYPm6VV3jXYGRkAAADAaBhkAAAAAKNhkAEAAACMhkEGAAAAMBqDDjKq6sSqOm/InsDikTXAEGQNMARZA99p6DMyTkziTQjMmqwBhiBrgCHIGjjI0IOM85OcUlV7quoFA/cGFoesAYYga4AhyBo4yNA3n392klNba6cdbceq2pVkV5JszbZZrwtYX2QNMARZAwxB1sBBjtmLfbbWdrfWdrbWdm7OlnkvB1inZA0wBFkDDEHWsCiO2UEGAAAAwMGGHmRcm2THwD2BxSNrgCHIGmAIsgYOMuggo7V2dZKLq+pyF6oBZkXWAEOQNcAQZA18p6Ev9pnW2hOG7gksHlkDDEHWAEOQNfDtXCMDAAAAGA2DDAAAAGA0Bv9qCQum+mZlf/C2l3fVP/mOD+iq7/H/2vVz6w2LZtNnvtRV/xvvfV1X/TNOPrOr/i5vmbz232QNrFptqL4DXHV1V/mbH3FhV/3P7bp/V/2mf5789df+r3f1BlavNvf9mf6qd7+6q/7sk87oqu/xlVV+rnFGBgAAADAaBhkAAADAaBhkAAAAAKNhkAEAAACMxmCDjKr61ap65lD9gMUka4AhyBpgCLIGDs0ZGQAAAMBozHSQUVW/XFUfr6q/T3LX5W2nVdV7q+rSqnptVd1ilmsA1j9ZAwxB1gBDkDVwdDMbZFTV6Ukel+ReSR6T5D7LT/1Jkl9srd0zyWVJ/vdh6ndV1SVVdcmN2TurZQIjJ2uAIcgaYAiyBlZnlmdkPDDJa1trX2+tfS3JG5LcLMmJrbV3LO9zYZIHHaq4tba7tbaztbZzc7bMcJnAyMkaYAiyBhiCrIFVmPU1MtqMjw+QyBpgGLIGGIKsgaOY5SDjnUnOrqrjq2pHkkcnuT7JNVX1wOV9npjkHYc7AMAqyBpgCLIGGIKsgVXYNKsDt9Y+VFWvTLInyWeTvGv5qZ9M8qKq2pbk00l+alZrANY/WQMMQdYAQ5A1sDozG2QkSWvteUmed4in7jvLvsBikTXAEGQNMARZA0c362tkAAAAAEyNQQYAAAAwGjP9aslUbdg4eenxW7ta77/++q76Rbb05S931T/5jg/oqv/MK+/ZVb/tndsnrt33yvd29QZWb98Xv9RV/4yTz+yqv+L5ffU3nrA0ce3e57+nqzcjVNVX3xb3hghtafL3WpLUCTu66p96yoO76j99/n266o/798l/d258qayBobS9e7vqzz7pjK76+ofv7qr/6kvvMHHt0l+t7m8oZ2QAAAAAo2GQAQAAAIyGQQYAAAAwGgYZAAAAwGgYZAAAAACjYZABAAAAjIZBBgAAADAam+a9gMOpql1JdiXJ1myb82qA9UrWAEOQNcAQZA2L4pg9I6O1tru1trO1tnNztsx7OcA6JWuAIcgaYAiyhkVxzA4yAAAAAA5mkAEAAACMxtwHGVX1kqraOe91AOubrAGGIGuAWZMzcAxc7LO19qR5rwFY/2QNMARZA8yanIFj4IwMAAAAgNUyyAAAAABGo1pr817DUVXVl5N89gi73CrJVzpaLHL9mNc+7/qj1d6xtXbrCY/NHMiaY7b32Otn3VvWjMw6z5oxr33e9cf62mXNyBzjWXOs/74fy/VjXvtq6leVNaMYZBxNVV3SWpv4gjeLXD/mtc+7vrc34zPm39fe+jGvfd7181474zPv35kx/76PuX7Ma2ec/L6Ps37Ma59G/U18tQQAAAAYDYMMAAAAYDTWyyBjt/pR9h57fW9vxmfMv6+99WNe+7zr5712xmfevzNj/n0fc/2Y1844+X0fZ/2Y1z6N+iTr5BoZi6yqrmutbV/x87lJdrbWnjKFY789yTNba5cctP2CJA9O8u/Lm85tre3p7Qccm+aUM5Xk/yT58SRLSV7YWvu93n7AsWtOWfOuJDuWf7xNkve31n60tx9w7JpT1jwsyQty4ESC63Lg76dP9vZbZJvmvQBG61mttVfPexHAunVukjskuVtrbX9V3WbO6wHWodbaA296XFWvSfL6OS4HWL9emOS/t9Y+WlXnJfmVHPisw4TWy1dLOISqunVVvaaqPrD8v/svbz+jqv6xqj68/N+7Lm8/vqr+vKourapXJjl+ri8AOObNMGd+Jsmvtdb2J0lr7apBXhBwTJr1Z5qq2pHkoUleN/MXAxyzZpg1LcnNlx+fkOQLM38x65wzMsbv+Kpa+bWOWyZ5w/Lj303yO621d1fV9yR5c5LvT/KxJA9qre2rqocn+fUkP5YDfzh8vbV2z6q6Z5IPHaHv86rquUnemuTZrbW9031ZwDFkHjlzSpLHVtXZSb6c5KmttU9M/ZUBx5J5faZJkrOTvLW19rUpvh7g2DSPrHlSkr+uqm8k+VqS+079VS0Yg4zx+0Zr7bSbfrjpO17LPz48yd0PfNU8SXLz5X9xOCHJhVV1lxyYDm5efv5BSX4vSVprl1bVpYfp+UtJvpjkuBy4WMsvJvm1ab0g4Jgzj5zZkuSbrbWdVfWYJC9N8sDD7AusD/PImps8PslLpvEigGPePLLm6Un+a2vtfVX1rCS/nQPDDSZkkLG+bUhyZmvtGys3VtXvJ3lba+3sqjo5ydtXPH3Uq7+21v51+eHeqnpZkmdOZbXAGM0kZ5JcmeQ1y49fm+Rl3SsFxmxWWZOq+q4kZ+TAWRnAYpt61lTVrZP8YGvtfcubXpnkb6e14EXlGhnr21uS/MfVd6vqpsnjCUk+v/z43BX7vzPJTyzve2qSex7qoFV1u+X/VpIfTXL5NBcNjMpMciYHvqf+0OXHD07yL9NZLjBSs8qa5MDdkd7UWvvmtBYLjNYssuaaJCdU1fct//yfk3x0ekteTAYZ69tTk+xcvvjMPyd58vL230jy/Kq6OMnGFfu/MMn25VOifiHJ+w9z3Iuq6rIklyW5VQ7cIhFYTLPKmfOT/Nhy1jw/Tr+ERTerrEmSxyX5sxmsGRifqWdNa21fkp9O8pqq+kiSJyZ51gxfw0Ko1lZ11h0AAADA3DkjAwAAABgNgwwAAABgNAwyAAAAgNEwyAAAAABGwyADAAAAGA2DDAAAAGA0DDIAAACA0fj/72mqBAiOrF4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1152x576 with 8 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Real translation: this is the first book i've ever done.\n"
     ]
    }
   ],
   "source": [
    "translate(\"este é o primeiro livro que eu fiz.\", plot='decoder_layer4_block2')\n",
    "print (\"Real translation: this is the first book i've ever done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RqQ1fIsLwkGE"
   },
   "source": [
    "## 总结\n",
    "\n",
    "在本教程中，您已经学习了位置编码，多头注意力，遮挡的重要性以及如何创建一个 transformer。\n",
    "\n",
    "尝试使用一个不同的数据集来训练 transformer。您可也可以通过修改上述的超参数来创建基础 transformer 或者 transformer XL。您也可以使用这里定义的层来创建 [BERT](https://arxiv.org/abs/1810.04805) 并训练最先进的模型。此外，您可以实现 beam search 得到更好的预测。"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "s_qNSzzyaCbD"
   ],
   "name": "transformer.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "001a7c0be8614db5912e0d84a340dc9f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "00c1711ff1c440a790c46c3df2ad2231": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_285bcddb37c54ff58f66102e9fcc24cd",
        "IPY_MODEL_ed8dd13e909f4231a2aedf431b6fa5bf"
       ],
       "layout": "IPY_MODEL_7c1fe99beeb847dc8228d68b96c026ae"
      }
     },
     "03bed3d858c346f6b0cc90c67ac83b48": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_5e8a0284070d434982478bcea9ee7821",
       "placeholder": "​",
       "style": "IPY_MODEL_8c8a8bf1a2c04c86b04f82ccdaf64b49",
       "value": " 1803/? [00:00&lt;00:00, 6998.75 examples/s]"
      }
     },
     "07c9a60ae4544a8da9c05d78d3c27355": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "0ac21e0c51a24e5c963d06738e8347dd": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "0e2bff89c2b1410a8de5a2afd8421188": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "109e553a1487413aab54ae95ab0ecbf9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Shuffling...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_5b3487117f3e44ff84adf5e30e454ca6",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_0ac21e0c51a24e5c963d06738e8347dd",
       "value": 1
      }
     },
     "124fa1f7cda0432682779318f91a1937": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "126772f0a9364369a4bc8daad5f5563e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_ddc80066957f4864a9bb029ee44ad369",
       "placeholder": "​",
       "style": "IPY_MODEL_6239d67352004e8c990334fcf6e25f6d",
       "value": " 1/1 [00:14&lt;00:00, 14.61s/ url]"
      }
     },
     "13827b7e891a49df9dab7ba83ebeeff8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "17a08b75cc1944c7b7abbbc485711c43": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_d3ccdbe33b5f4ff68a6114b4a3bf419c",
        "IPY_MODEL_03bed3d858c346f6b0cc90c67ac83b48"
       ],
       "layout": "IPY_MODEL_b1c649f863f1484b894aca973eabbfab"
      }
     },
     "1847a58007604463bbcb6849e393e337": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_6ff14ff33c694bca836ed17a1d7fd8d3",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_988fbcb9b7de4a8ba6c15907d6b6e8e8",
       "value": 1
      }
     },
     "19d30031b02a49e4927d74250be77080": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "1bbb97d9c39749e980ce979c45a80d90": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_538584a538ab4d2496eb3702f72f8fa5",
        "IPY_MODEL_3151d5dcde83422281cfae5f75d1caf8"
       ],
       "layout": "IPY_MODEL_936b49d5b53a4975b6f8e40ea96f1d9c"
      }
     },
     "1da50d0eb8e64ccd961c90d35fe679f0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "1ded2b5cc1c24b48bf2b811064c3aacb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "Reading...: ",
       "description_tooltip": null,
       "layout": "IPY_MODEL_838e318bc5914382b2c6af79bca7041a",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_edcce3dfbf8e4171b49e8e4efa53b341",
       "value": 1
      }
     },
     "205653928d9f4f429830054f7a25745f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "2544e5a7552d45cda639e0a78d032927": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "26c85d9783cd41408c0cfbaee9383d25": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "285bcddb37c54ff58f66102e9fcc24cd": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "Reading...: ",
       "description_tooltip": null,
       "layout": "IPY_MODEL_07c9a60ae4544a8da9c05d78d3c27355",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_d5a0d54fea3e4d5f9762b4b25fd447d9",
       "value": 1
      }
     },
     "289419694f8045c78fdb9fc51a6f5f93": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_a2a2178b621940c2a69c2bebbfad02ed",
        "IPY_MODEL_f986b86d964346bdb30d34a1a8c22135"
       ],
       "layout": "IPY_MODEL_2e18882dd3814f058cd8794e601345f6"
      }
     },
     "2ba2a131d04a412db8689f23e604d756": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "2d0c9cab9b5c417abe2d0570e0a0f5f4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "2e18882dd3814f058cd8794e601345f6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "2e546bb9bf8e4f1e9fe92dc565327e20": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "30214e97771c4c2081440b834242724b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "30ac003d70da4b1b974b47fca60263a7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Writing...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_124fa1f7cda0432682779318f91a1937",
       "max": 1193,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_36a2ad95f9414c95a452fc3da83edcdf",
       "value": 1193
      }
     },
     "3151d5dcde83422281cfae5f75d1caf8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_e4c6c4355fca47af9672f5c4afbada7c",
       "placeholder": "​",
       "style": "IPY_MODEL_4d38bff319444319947c77ad9d39f249",
       "value": " 1803/1803 [00:00&lt;00:00, 91795.90 examples/s]"
      }
     },
     "33d49bb4a4574f209dd105fe9cac24ca": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "35982fc018614bca8608acf189d3b525": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_86295548d36b49f2aa831ff3c98ce984",
       "placeholder": "​",
       "style": "IPY_MODEL_54f7a00efe364fad986a2775d8ddb744",
       "value": " 51785/? [00:00&lt;00:00, 527579.96 examples/s]"
      }
     },
     "36a2ad95f9414c95a452fc3da83edcdf": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "398409de76504ef7956a48882441ec0b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "3af98c2036d045f0a6e91eb255c1e0b1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "3b447576b8e545dea94cd79a04e563c8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "3e968be17e674780be775ea5939b3713": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_dd8c2c6b63eb4c238c4456019473be4b",
        "IPY_MODEL_35982fc018614bca8608acf189d3b525"
       ],
       "layout": "IPY_MODEL_55069a65c7a046c784ac42f246e81940"
      }
     },
     "4668d7d45e424f6ba128b7764efd4d6e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "48545b15af944667b6e4e0f5f7c8c941": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "4b25fe2619c943779b83ce65ffc1d81f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "4d38bff319444319947c77ad9d39f249": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "4d8f49b1bdd9402c9279233f94ea9fdf": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "538584a538ab4d2496eb3702f72f8fa5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Writing...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_c7ed38a58b634cc7a7272551c7cfdfd6",
       "max": 1803,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_001a7c0be8614db5912e0d84a340dc9f",
       "value": 1803
      }
     },
     "54f7a00efe364fad986a2775d8ddb744": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "55069a65c7a046c784ac42f246e81940": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "56cb634112c049d9b70d3cb8b5d50d06": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "56d4618b01aa4dbc8c54d6fb13108dd1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_95f595f32a1e46e4b4d4434202db7b39",
       "placeholder": "​",
       "style": "IPY_MODEL_cb6eb0cca67c4787b77a9479b5fd5a1b",
       "value": " 1/1 [00:14&lt;00:00, 14.57s/ file]"
      }
     },
     "581ac240edb84cb4975ec8cfd28f49fc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "5b3487117f3e44ff84adf5e30e454ca6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "5d3127e13ec84b558fa092f0d7819bb7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "5d35841e8c0f4a2aba7c32951ac3449c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_cd6cf105757e4214ba1828547aa61510",
        "IPY_MODEL_a75a27d33d6e43368abedd83346674bc"
       ],
       "layout": "IPY_MODEL_8ef16492a6e14c98a1f476d506906aa3"
      }
     },
     "5dd4da35e8e44bd29abfa75bdb5476d6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "5e364761de6c4117859ea4b1e096bbf2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_6b1448a0807842b9a689e71a4f0e3186",
        "IPY_MODEL_c3316461d2df45a28e07d5e52cfbaf0d"
       ],
       "layout": "IPY_MODEL_bc69556f0dcf481dac2c7b394efdaec6"
      }
     },
     "5e8a0284070d434982478bcea9ee7821": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "6239d67352004e8c990334fcf6e25f6d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "64661e1df102453a92b8799360c7f6c2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_30214e97771c4c2081440b834242724b",
       "placeholder": "​",
       "style": "IPY_MODEL_48545b15af944667b6e4e0f5f7c8c941",
       "value": " 1/1 [00:00&lt;00:00, 17.16 shard/s]"
      }
     },
     "66cffae977f6483aa93f3fd6246a4bac": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_96f61db07ef94c629438b3a765d24fb3",
       "placeholder": "​",
       "style": "IPY_MODEL_4d8f49b1bdd9402c9279233f94ea9fdf",
       "value": " 1193/1193 [00:00&lt;00:00, 68294.54 examples/s]"
      }
     },
     "6b1448a0807842b9a689e71a4f0e3186": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Writing...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_9f8026bd5b624ed38e5e67f93e9dc464",
       "max": 51785,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_2d0c9cab9b5c417abe2d0570e0a0f5f4",
       "value": 51785
      }
     },
     "6ce00f263f0c4985bc490059d7bf6b37": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_76a09b996120403e9beaefbc4fbd12a1",
        "IPY_MODEL_9497b5de2e5447f288047614080c7897"
       ],
       "layout": "IPY_MODEL_f2a27a3e607240e1a4143a6f4873d9b1"
      }
     },
     "6ce7314b9588431db6d7bc699938f723": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_efc7ed3db3274733be32c8317459f71c",
        "IPY_MODEL_126772f0a9364369a4bc8daad5f5563e"
       ],
       "layout": "IPY_MODEL_73708d36bc8048e18cd3ccf899acbbbb"
      }
     },
     "6ff14ff33c694bca836ed17a1d7fd8d3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "71f77badfbfb41d4a8d937961a7d2943": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "73708d36bc8048e18cd3ccf899acbbbb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "739f70db5a814d048a05ed6aa5e3cde1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "76a09b996120403e9beaefbc4fbd12a1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Dl Size...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_581ac240edb84cb4975ec8cfd28f49fc",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_2544e5a7552d45cda639e0a78d032927",
       "value": 1
      }
     },
     "7c1fe99beeb847dc8228d68b96c026ae": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "7ca48a7537804728a56bed9e59e76626": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "838e318bc5914382b2c6af79bca7041a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "86295548d36b49f2aa831ff3c98ce984": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "882aab6d1a244a68a265580a32af36b2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_109e553a1487413aab54ae95ab0ecbf9",
        "IPY_MODEL_f7750280f4704c7c803e24581a618128"
       ],
       "layout": "IPY_MODEL_bf2fcf010136467bafbbdb533e0c5688"
      }
     },
     "8c8a8bf1a2c04c86b04f82ccdaf64b49": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "8ef16492a6e14c98a1f476d506906aa3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "8fd458ed5cef4298b322846d32f0929d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Shuffling...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_19d30031b02a49e4927d74250be77080",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_c707c4b49fcb43cfa45f4c7ba1e97e47",
       "value": 1
      }
     },
     "936b49d5b53a4975b6f8e40ea96f1d9c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "9497b5de2e5447f288047614080c7897": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_2ba2a131d04a412db8689f23e604d756",
       "placeholder": "​",
       "style": "IPY_MODEL_3af98c2036d045f0a6e91eb255c1e0b1",
       "value": " 124/124 [00:14&lt;00:00,  8.50 MiB/s]"
      }
     },
     "95f595f32a1e46e4b4d4434202db7b39": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "96f61db07ef94c629438b3a765d24fb3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "988fbcb9b7de4a8ba6c15907d6b6e8e8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "99c47f38dc144ece80991a203ccbca7f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_1847a58007604463bbcb6849e393e337",
        "IPY_MODEL_c07f3dd9e60342779cc4c1a08c7d6f92"
       ],
       "layout": "IPY_MODEL_5d3127e13ec84b558fa092f0d7819bb7"
      }
     },
     "9b57950023f04dd392a9df2bf263ac03": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_30ac003d70da4b1b974b47fca60263a7",
        "IPY_MODEL_66cffae977f6483aa93f3fd6246a4bac"
       ],
       "layout": "IPY_MODEL_739f70db5a814d048a05ed6aa5e3cde1"
      }
     },
     "9e99e92143aa44f1ba434c3f7649d2b2": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "9ec4bd8bf7df4450a2f5f5616e344f72": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "9f8026bd5b624ed38e5e67f93e9dc464": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "a2a2178b621940c2a69c2bebbfad02ed": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_4668d7d45e424f6ba128b7764efd4d6e",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_4b25fe2619c943779b83ce65ffc1d81f",
       "value": 1
      }
     },
     "a5b1b5f834f845c3bf6eabb2b4d08306": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "a75a27d33d6e43368abedd83346674bc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_13827b7e891a49df9dab7ba83ebeeff8",
       "placeholder": "​",
       "style": "IPY_MODEL_e3543424ed374827ba60bfa73d17553b",
       "value": " 1/1 [00:00&lt;00:00,  3.18 shard/s]"
      }
     },
     "a7daf30abca8421392212c589fa61887": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "aa5fcad150224b49af8f197e5ce02552": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b1c649f863f1484b894aca973eabbfab": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b32d5689be4443ba9ffd3f012acc3493": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b5523cd2bdc64a8ea90005aef2cb0a30": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_398409de76504ef7956a48882441ec0b",
       "placeholder": "​",
       "style": "IPY_MODEL_3b447576b8e545dea94cd79a04e563c8",
       "value": " 1803/? [00:00&lt;00:00, 85565.11 examples/s]"
      }
     },
     "ba6f07e57ad547569002b98431c5a27a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bae085d8f85b47f9b7fdd5f6353f64ac": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bc69556f0dcf481dac2c7b394efdaec6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bf2fcf010136467bafbbdb533e0c5688": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "c07f3dd9e60342779cc4c1a08c7d6f92": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_b32d5689be4443ba9ffd3f012acc3493",
       "placeholder": "​",
       "style": "IPY_MODEL_0e2bff89c2b1410a8de5a2afd8421188",
       "value": " 1193/? [00:00&lt;00:00, 6449.96 examples/s]"
      }
     },
     "c3316461d2df45a28e07d5e52cfbaf0d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_fc4540bc47ef4073a335e50336555239",
       "placeholder": "​",
       "style": "IPY_MODEL_26c85d9783cd41408c0cfbaee9383d25",
       "value": " 51785/51785 [00:00&lt;00:00, 284136.21 examples/s]"
      }
     },
     "c45eb40a3b9b4d4d8392f744a99a3b16": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_8fd458ed5cef4298b322846d32f0929d",
        "IPY_MODEL_64661e1df102453a92b8799360c7f6c2"
       ],
       "layout": "IPY_MODEL_a5b1b5f834f845c3bf6eabb2b4d08306"
      }
     },
     "c707c4b49fcb43cfa45f4c7ba1e97e47": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "c7ed38a58b634cc7a7272551c7cfdfd6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "cb6eb0cca67c4787b77a9479b5fd5a1b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "cd6cf105757e4214ba1828547aa61510": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Shuffling...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_9ec4bd8bf7df4450a2f5f5616e344f72",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_1da50d0eb8e64ccd961c90d35fe679f0",
       "value": 1
      }
     },
     "d2786b06c84b4854a1f65d230bd7743d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Extraction completed...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_56cb634112c049d9b70d3cb8b5d50d06",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_205653928d9f4f429830054f7a25745f",
       "value": 1
      }
     },
     "d3ccdbe33b5f4ff68a6114b4a3bf419c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_a7daf30abca8421392212c589fa61887",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_7ca48a7537804728a56bed9e59e76626",
       "value": 1
      }
     },
     "d3fe901234504391b426634ba6c606eb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d5a0d54fea3e4d5f9762b4b25fd447d9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "dd8c2c6b63eb4c238c4456019473be4b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "Reading...: ",
       "description_tooltip": null,
       "layout": "IPY_MODEL_ba6f07e57ad547569002b98431c5a27a",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_2e546bb9bf8e4f1e9fe92dc565327e20",
       "value": 1
      }
     },
     "ddc80066957f4864a9bb029ee44ad369": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e3543424ed374827ba60bfa73d17553b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "e4c6c4355fca47af9672f5c4afbada7c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e83c4da4c14044b48a45954aee03ddbc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_d2786b06c84b4854a1f65d230bd7743d",
        "IPY_MODEL_56d4618b01aa4dbc8c54d6fb13108dd1"
       ],
       "layout": "IPY_MODEL_d3fe901234504391b426634ba6c606eb"
      }
     },
     "ed8dd13e909f4231a2aedf431b6fa5bf": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_f06a591701384752a8d7182d64af5990",
       "placeholder": "​",
       "style": "IPY_MODEL_33d49bb4a4574f209dd105fe9cac24ca",
       "value": " 1193/? [00:00&lt;00:00, 60775.20 examples/s]"
      }
     },
     "edcce3dfbf8e4171b49e8e4efa53b341": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "ee572793477d447aa74db198a0928b6a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_1ded2b5cc1c24b48bf2b811064c3aacb",
        "IPY_MODEL_b5523cd2bdc64a8ea90005aef2cb0a30"
       ],
       "layout": "IPY_MODEL_aa5fcad150224b49af8f197e5ce02552"
      }
     },
     "efc7ed3db3274733be32c8317459f71c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Dl Completed...: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_fe97e0a014064417b2ba12d523d1092b",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_f52df524204a4f9a8b477a08dafc211d",
       "value": 1
      }
     },
     "f06a591701384752a8d7182d64af5990": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "f2a27a3e607240e1a4143a6f4873d9b1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "f52df524204a4f9a8b477a08dafc211d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "f7750280f4704c7c803e24581a618128": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_bae085d8f85b47f9b7fdd5f6353f64ac",
       "placeholder": "​",
       "style": "IPY_MODEL_71f77badfbfb41d4a8d937961a7d2943",
       "value": " 1/1 [00:00&lt;00:00, 18.45 shard/s]"
      }
     },
     "f986b86d964346bdb30d34a1a8c22135": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_9e99e92143aa44f1ba434c3f7649d2b2",
       "placeholder": "​",
       "style": "IPY_MODEL_5dd4da35e8e44bd29abfa75bdb5476d6",
       "value": " 51785/? [00:06&lt;00:00, 7673.46 examples/s]"
      }
     },
     "fc4540bc47ef4073a335e50336555239": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "fe97e0a014064417b2ba12d523d1092b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
